Methods and apparatus for using multiple optical chains in parallel to support separate color-capture

ABSTRACT

Methods and apparatus for capturing or generating images using multiple optical chains operating in parallel are described. Pixel values captured by individual optical chains corresponding to the same scene area are combined to provide an image with at least some of the benefits which would have been provided by capturing an image of the scene using a larger lens than that of the individual lenses of the optical chain modules. By using multiple optical chains in parallel at least some benefits normally obtained from using a large lens can be obtained without the need for a large lens. Furthermore in at least some embodiments, a wide dynamic range can be supported through the use of multiple sensors with the overall supported dynamic range being potentially larger than that of the individual sensors. Some lens and/or optical chain configurations are designed for use in small handheld devices, e.g., cell phones.

RELATED APPLICATIONS

The present application claims the benefit of U.S. Provisional PatentApplication Ser. No. 61/749,314, filed Jan. 5, 2013, U.S. ProvisionalPatent Application Ser. No. 61/749,315, filed Jan. 5, 2013, U.S.Provisional Patent Application Ser. No. 61/749,316, filed Jan. 5, 2013,U.S. Provisional Patent Application Ser. No. 61/749,317, filed Jan. 5,2013, and U.S. Provisional Patent Application Ser. No. 61/749,382, filedJan. 6, 2013, and is related to U.S. Provisional Patent Application Ser.No. 61/923,755, filed Jan. 5, 2014 each of the forgoing patentapplications being hereby expressly incorporated by reference in theirentirety.

FIELD

The present application relates to image capture and generation methodsand apparatus and, more particularly, to methods and apparatus relatedto camera apparatus including multiple optical chains or which processesthe output of multiple optical chains.

BACKGROUND

High quality digital cameras have to a large extent replaced filmcameras. However, like film cameras, with digital cameras much attentionhas been placed by the camera industry on the size and quality of lenseswhich are used on the camera. Individuals seeking to take qualityphotographs are often encouraged to invest in large bulky and oftencostly lenses for a variety of reasons. Among the reasons for usinglarge aperture lenses is their ability to capture a large amount oflight in a given time period as compared to smaller aperture lenses.Telephoto lenses tend to be large not only because of their largeapertures but also because of their long focal lengths. Generally, thelonger the focal length, the larger the lens. A long focal length givesthe photographer the ability to take pictures from far away.

In the quest for high quality photos, the amount of light which can becaptured is often important to the final image quality. Having a largeaperture lens allows a large amount of light to be captured allowing forshorter exposure times than would be required to capture the same amountof light using a small lens. The use of short exposure times can reduceblurriness especially with regard to images with motion. The ability tocapture large amounts of light can also facilitate the taking of qualityimages even in low light conditions. In addition, using a large aperturelens makes it possible to have artistic effects such as small depth offield for portrait photography.

Large lenses sometimes also offer the opportunity to support mechanicalzoom features that allow a user to optically zoom in or out and/or toalter the focal length of the lens which is important for framing ascene without the need to move closer or further from the subject.

While large lenses have many advantages with regard to the ability tocapture relatively large amounts of light compared to smaller lenses,support large zoom ranges, and often allow for good control over focus,there are many disadvantages to using large lenses.

Large lenses tend to be heavy requiring relatively strong and oftenlarge support structures to keep the various lenses of a camera assemblyin alignment. The heavy weight of large lenses makes cameras with suchlenses difficult and bulky to transport. Furthermore, cameras with largelenses often need a tripod or other support to be used for extendedperiods of time given that the sheer weight of a camera with a largelens can become tiresome for an individual to hold in a short amount oftime.

In addition to weight and size drawbacks, large lenses also have thedisadvantage of being costly. This is because of, among other things,the difficultly in manufacturing large high quality optics and packagingthem in a manner in which they will maintain proper alignment over aperiod of time which may reflect the many years of use a camera lensesis expected to provide.

A great deal of effort has been directed in the camera industry tosupporting the use of large camera lenses and packaging them in a waythat allows different lenses to be used in an interchangeable manner ona camera body. However, for the vast majority of camera users, thedrawbacks to cameras with large lenses means that camera users tend notto use large lenses with such lenses often being left to professionalsand/or photo enthusiasts willing to incur the expense and trouble ofbuying and using large lenses.

In fact, many camera owners who own cameras with large high qualitylenses often find themselves taking pictures with small pocket sizecameras, often integrated into other devices such as their cell phones,personal digital assistants or the like, simply because they are moreconvenient to carry. For example, cell phone mounted cameras are oftenmore readily available for use when an unexpected photo opportunityarises or in the case of a general family outing where carrying largebulky camera equipment may be uncomfortable or undesirable.

To frame a given scene from a given point, the focal length (hence size)of the lens depends on the size (area) of the image sensor. The smallerthe image sensor, the smaller the focal length and the smaller the lensrequired. With advances in sensor technology, it is now possible to makesmall sensors, e.g., 5×7 mm² sensors, with relatively high pixel count,e.g., 8 megapixels. This has enabled the embedding of relatively highresolution cameras in small devices such as cell phones. The smallsensor size (compared to larger cameras such as changeable lenssingle-lens reflex (SRL) cameras) enables small focal length lenseswhich are much smaller and lighter than larger focal length lensesrequired for cameras with larger sensors.

Cell phone mounted cameras and other pocket sized digital camerassometimes rely on a fixed focal length lens which is also sometimesreferred to as a focus-free lens. With such lenses the focus is set atthe time of manufacture, and remains fixed. Rather than having a methodof determining the correct focusing distance and setting the lens tothat focal point, a small aperture fixed-focus lens relies on a largedepth of field which is sufficient to produce acceptably sharp images.Many cameras, including those found on most cell phones, with focus freelenses also have relatively small apertures which provide a relativelylarge depth of field. There are also some high end cell phones that useauto focus cameras.

For a lens of a digital camera to be useful, it needs to be paired witha device which detects the light passing through the lens and convertsit to pixel (picture element) values. A megapixel (MP or Mpx) is onemillion pixels. The term is often used to indicate the number of pixelsin an image or to express the number of image sensor elements of adigital camera where each sensor element normally corresponds to onepixel. Multi-color pixels normally include one pixel value for each ofthe red, green, and blue pixel components.

In digital cameras, the photosensitive electronics used as the lightsensing device is often either a charge coupled device (CCD) or acomplementary metal oxide semiconductor (CMOS) image sensor, comprisinga large number of single sensor elements, each of which records ameasured intensity level.

In many digital cameras, the sensor array is covered with a patternedcolor filter mosaic having red, green, and blue regions in anarrangement. A Bayer filter mosaic is one well known a color filterarray (CFA) for arranging RGB color filters on a square grid of photosensors. Its particular arrangement of color filters is used in manydigital image sensors. In such a filter based approach to capturing acolor image, each sensor element can record the intensity of a singleprimary color of light. The camera then will normally interpolate thecolor information of neighboring sensor elements, through a processsometimes called demosaicing, to create the final image. The sensorelements in a sensor array using a color filter are often called“pixels”, even though they only record 1 channel (only red, or green, orblue) of the final color image due to the filter used over the sensorelement.

While a filter arrangement over a sensor array can be used to allowdifferent sensor elements to capture different colors of light thusallowing a single sensor to capture a color image, the need to carefullyalign the filter area with individual pixel size sensor elementscomplicates the manufacture of sensor arrays as compared to arrays whichdo not require the use of a multi-color filter array. Furthermore, thefact that multiple colors of light need to pass through the cameralenses to reach the sensor so that the sensor can measure multipledifferent colors of light means that the lens can not be optimized for asingle color of light and that some chromatic aberration is likely toresult. Chromatic aberration is a type of distortion in which there is afailure of a lens to focus all colors to the same convergence point. Itoccurs because lenses have a different refractive index for differentwavelengths of light sometimes referred to as the dispersion of thelens. While small focal length lenses paired with relatively highresolution sensors have achieved widespread commercial success in cellphones and pocket cameras, they often leave their owners longing forbetter picture quality, e.g., picture quality that can only be achievedwith a larger pixel area and a larger lens opening to collect morelight.

Smaller sensors require smaller focal length lenses (hence smallerlenses) to frame the same scene from the same point. Availability ofhigh pixel count small sensors means that a smaller lens can be used.However, there are a few disadvantages to using smaller sensors andlenses. First, the small pixel size limits the dynamic range of thesensor as only a small amount of light can saturate the sensor. Second,small lenses collect less total light which can result in grainypictures. Third, small lenses have small maximum apertures which makeartistic effects like small depth of field for portrait pictures notpossible.

In the case of a single lens being used to capture color images, thereare often problems associated with the use of a color filter maskbetween the lens and the image sensor. This is because the multi-colorgrid mask needs to be carefully aligned with the image sensor elementsfor proper color separation. While the use of a multi-color grid maskhas the benefit of allowing a single lens and image sensor to capturecolor images, the number of sensor elements per image area is reduced ona per color basis as compared to the case where the image sensor isdedicated to capturing light corresponding to a single color. Forexample, if an 18 MP sensor is used to evenly capture R, G and B light,on a per color basis each color would be captured at only a 6 Mpelresolution. In addition to resulting in a per color pixel resolutionthat is lower than the overall pixel resolution of the sensor, the costof camera modules using a multi-color mask tends to be more expensivethen the cost of monochrome, i.e., single color, camera modules usingmonochrome lens which do not require a grid like filter. From the abovediscussion it should be appreciated that the use of camera modules withmulti-color grid filters between a camera lens and photo sensor haveseveral disadvantages.

From the above, it should be appreciated that there is a need forimproved methods and apparatus for capturing and/or generating colorimages which avoid or reduce the need for image sensors with multi-colorfilters.

SUMMARY OF THE INVENTION

Methods and apparatus which use multiple optical chains to capturemultiple images of an area at the same time are described. The multiplecaptured images may, and in some embodiments are then combined to form acombined image. The combined image in various embodiments is normally ofhigher quality than would be achieved using just a single one of theoptical chains.

The use of multiple optical chains in parallel, in various embodiments,provides many of the benefits associated with use of a large lens and/orlarge high quality sensor, through the use of multiple optical chainswhich can normally be implemented using smaller and/or lower costcomponents than commonly used with a high quality large lens singleoptical chain camera implementation.

In various embodiments an optical chain includes a combination ofelements including one or more lenses and a sensor. The outer lenses ofthe optical chains in various embodiments have a known relationship,i.e., spacing, between them. This allows for pixels of an image capturedby one optical chain to be easily matched to and combined with pixelscorresponding to the same scene area captured by one or more of theother optical chains and then processed to generate a combined imagefrom the various images captured by the sensors of the individualoptical chains.

In various embodiments since one or more pixel values captured bydifferent sensors are combined to generate a pixel value of the combinedimage, the number of pixel values in the combined image in one suchembodiment will be equal to or less than the number of pixel values inthe individual images, captured by different optical chains,corresponding to the same scene area from which the combined image isgenerated. For example, if three 8 megapixel sensors are used inparallel to capture 3 images which are then used to generate a combinedimage, the combined image will, in some but not necessarily allembodiments, be of 8 megapixels or less. Thus, unlike the case whereimages are combined to generate a panoramic view with the panoramic viewincluding more pixels than any individual image used to generate thepanoramic image, in some embodiments the combined image is constrainedto having the same or fewer pixels than the highest resolution imageused as part of the combining process to generate the combined image.

However, when at least some of the images which are combined correspondto different scene areas the number of pixels in the resulting image maybe greater than that of an individual sensor since different sensorsprovide values which contribute to different scene areas. Thus, inembodiments of the present invention where a panoramic or other view isgenerated from images captured from optical chain modules which capturedifferent scene areas the number of pixels in the image is not limitedby the number of pixels in the sensor used by an individual opticalchain module.

The methods and apparatus of the invention allow, in some embodiments,different, e.g., relatively low cost, optical chains to be used inparallel to provide many of the benefits of a large lens, e.g., a largelight capture area, without the need for a large single large lens andmany of the disadvantages associated with a single large lens not onlyin terms of cost, weight, size and/or other issues.

In some but not all embodiments, short focal length lenses with eachlens having a sensor behind are used to capture a plurality of imagescorresponding to the same scene image area at the same time followed bycombining/processing of the images to form one or more images with equalor lower pixel counts than either of the individual captured images.

User control of focus and depth of field in the combined image can, andin some embodiments is, controlled via post image capture processing.While the focus of the individual optical chain modules may becontrolled in response to user input at the time of image capture,because multiple images are captured using separate optical chains and aknown physical separation between the lenses, the focus point of thecombined image generated from images captured in parallel by themultiple optical chains can, and in various embodiments is, controlledby user input used to control how the images are combined after they arecaptured to produce the combined image.

Various methods and apparatus of the present invention provide some orall of the benefits of using relatively large and/or long lensassemblies without the need for large lens and/or long lens assemblies,through the use of multiple optical chain modules in combination.

While the methods and apparatus support the capture and generation ofindividual images at point in time, they can also be used to capturevideo. Thus, while some embodiments are directed to camera devices whichcapture still images, other embodiments are directed to camera devicewhich do capture video and/or still images using multiple optical chainsoperating in parallel.

Using the methods and apparatus of the present invention, a handheldcamera can provide improved still image and/or video generation resultsthan might be achieved without the use of the methods described herein.

In some embodiments, optical chain modules use relatively short focallength lenses, e.g., of the type commonly used in cell phones, whichrequire relatively little depth (thickness) within a camera as comparedto larger lens cameras. This allows for a camera device implemented inaccordance with some embodiments to be relatively thin and still provideat least some of the benefits normally provided by much thicker lenseswhich thus require a greater overall camera thickness than is requiredby various embodiments described herein.

While use of short focal length lens can have advantages in terms ofsmall lens thickness, the methods and apparatus of the present are notlimited to the use of such lenses and can be used with a wide variety oflens types. In addition, while numerous embodiments are directed toautofocus embodiments, fixed focus embodiments are also possible andsupported.

An optical chain, in various embodiments, includes a first lens and animage sensor. Additional lenses and/or one or more optical filters maybe included between the first lens of an optical chain module and theimage sensor depending on the particular embodiment. In some cases theremay be one or more optical filters before the first lens.

The use of multiple optical chain modules is well suited for use indevices such as cell phones and/or portable camera devices intended tohave a thin form factor, e.g., thin enough to place in a pocket orpurse. By using multiple optical chains and then combining the capturedimages or portions of the captured images to produce a combined image,improved images are produced as compared to the case where a singleoptical chain module of the same size is used.

While in various embodiments separate image sensors are used for each ofthe individual optical chain modules, in some embodiments the imagesensor of an individual optical chain module is a portion of a CCD orother optical sensor dedicated to the individual optical chain modulewith different portions of the same sensor serving as the image sensorsof different optical chain modules.

In various embodiments, images of a scene area are captured by differentoptical chain modules and then subsequently combined either by theprocessor included in the camera device which captured the images or byanother device, e.g., a personal or other computer which processes theimages captured by the multiple optical chains after offloading from thecamera device which captured the images.

At least some aspects of the present invention relate to using opticalchains corresponding to individual colors. In this manner the fullresolution of a sensor can be dedicated to a single sensor allowing fora higher overall resolution than if a sensor is divided for use tocapture multiple different sensors, e.g., at ⅓ the resolution of theoverall sensor.

Furthermore, by dedicating individual optical chains to a single color,the components for the individual chain can be optimized for the colorwithout concern for chromatic aberrations that a lens can introduce whenthe same lens is used to capture multiple colors.

By using different optical chains for different colors, implementationis simplified, chromatic aberration can be reduced and/or more easilycorrected and/or other benefits are possible.

While individual colors may be captured by different optical chains, insome embodiments at least one sensor with a Bayer or other color filteris included to support preview operation and/or facilitate focus settingby a user without the need to generate a combined image from the pixelvalues captured by multiple optical chains.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1A is an exemplary block diagram of an exemplary apparatus, e.g.,camera device, implemented in accordance with one embodiment of thepresent invention.

FIG. 1B illustrates a frontal view of an apparatus implemented inaccordance with an exemplary embodiment of the present invention whichincorporates multiple optical chain modules in accordance with thepresent invention with lenses which are viewable from the front of thecamera.

FIG. 1C, which is a side view of the exemplary apparatus of FIG. 1B,illustrates further details of the exemplary apparatus.

FIG. 1D illustrates a plurality of optical chain modules that can beused in an exemplary device implemented in accordance with theinvention.

FIG. 2 illustrates a camera device implemented in accordance with oneembodiment of the present invention.

FIG. 3A shows an exemplary lens configuration which may be used for theset of outer lenses of the camera device shown in FIGS. 1A-1C.

FIG. 3B illustrates an exemplary filter arrangement which is used in thecamera of FIGS. 1A-1C in some embodiments.

FIG. 3C shows an exemplary inner lens configuration which may, and insome embodiments is, used for a set of inner lenses of the camera deviceshown in FIGS. 1A-1C.

FIG. 4 illustrates an exemplary camera device in which the sets of outerlenses, filters, and inner lenses are mounted on corresponding platters.

FIG. 5A illustrates various filter and lens platters that may be used inthe camera device shown in FIG. 4 depending on the particularembodiment.

FIG. 5B illustrates the filter platter arrangement shown in FIG. 5A whenviewed from the side and when viewed from the front.

FIG. 6, which comprises the combination of FIGS. 6A, 6B, and 6C, showsan exemplary combination of lenses and filters used in one exemplaryembodiment in which a single color filter is used in at least some ofthe different optical chain modules.

FIG. 7, which comprises the combination of FIGS. 7A, 7B, and 7C, showsan exemplary combination of lenses and filters used in one exemplaryembodiment in which exposures of different duration are used fordifferent optical chain modules and a single color filter is used in atleast some of the different optical chain modules.

FIG. 8 illustrates an optical chain arrangement used in one panoramiccamera embodiment in which multiple optical chains and different lensangles are used to capture images that are well suited for combininginto a panoramic image.

FIG. 9 illustrates an exemplary method of producing at least one imageof a first scene area by operating a plurality of optical chain modulesin accordance with one embodiment of the present invention.

FIG. 10 illustrates an exemplary method of producing at least one imageof a first scene area with an enhanced sensor dynamic range by operatingtwo or more optical chain modules in accordance with one embodiment ofthe present invention.

FIG. 11 illustrates an exemplary method of producing at least one imageof a first scene area with enhanced sensor dynamic range by operatingtwo or more optical chain modules in accordance with one embodiment ofthe present invention.

FIG. 12 illustrates an exemplary method of producing at least one imageof a first scene area by operating two or more optical chain modulesusing color filters in accordance with one embodiment of the presentinvention.

FIG. 13 illustrates an exemplary assembly of modules, which may, and insome embodiments is, part of an apparatus which implements one or moremethods of the invention, for performing various image and dataprocessing functions in accordance with one or more exemplaryembodiments of the invention.

FIG. 14 illustrates a computer system which can be used for postprocessing of images captured using a camera device.

FIG. 15 illustrates a frontal view of an apparatus implemented inaccordance with one embodiment of the present invention whichincorporates multiple optical chain modules, e.g., one for each of red,green and blue and one for all three colors.

FIG. 16 illustrates a frontal view of the outer lens assembly of anapparatus implemented in accordance with one embodiment of the presentinvention where the apparatus incorporates multiple optical chainmodules and outer lenses configured with little or no gaps between thelenses.

FIG. 17 illustrates a frontal view of the outer lenses of a lensassembly implemented in accordance with one embodiment of the presentinvention where the apparatus incorporates multiple optical chainmodules with lenses configured with little or no gaps between the lensesbut non-uniform spacing between the optical centers of at least some ofthe lenses.

FIG. 18 illustrates a camera device including a plurality of opticalchain modules which includes mirrors or another device for changing theangle of light entering the optical chain module and thereby allowing atleast a portion of the optical chain module to extend in a direction,e.g., a perpendicular direction, which is not a straight front to backdirection with respect to the camera device.

FIG. 19 illustrates another camera device including a plurality ofoptical chain modules which includes mirrors or another device forchanging the angle of light entering the optical chain module andthereby allowing at least a portion of the optical chain module toextend in a direction, e.g., a perpendicular direction, which is not astraight front to back direction with respect to the camera device.

FIG. 20 is a drawing of an assembly of modules, which may be included inan exemplary apparatus, e.g., a camera device, in accordance with anexemplary embodiment.

FIG. 21 is a drawing of an assembly of modules, which may be included inan exemplary apparatus, e.g., a camera device, in accordance with anexemplary embodiment.

FIG. 22 is a drawing of an assembly of modules, which may be included inan exemplary apparatus, e.g., a camera device, in accordance with anexemplary embodiment.

FIG. 23 is a drawing of an assembly of modules, which may be included inan exemplary apparatus, e.g., a camera device, in accordance with anexemplary embodiment.

FIG. 24A is a first part of FIG. 24 which shows a method of generating acombined image from pixel values generated by multiple optical chainmodules operating in parallel.

FIG. 24B is a second part of FIG. 24 which shows a method of generatinga combined image from pixel values generated by multiple optical chainmodules operating in parallel.

FIG. 24 is the combination of FIG. 24A and FIG. 24B and shows shows amethod of generating a combined image from pixel values generated bymultiple optical chain modules operating in parallel.

FIG. 25 illustrates a chart illustrating pixel values being arrangedinto sets corresponding to different pixel areas in accordance with themethod shown in FIG. 24.

FIG. 26 illustrates a chart of captured pixel values with pixel valueswhich will be excluded from use due to there correspondence to asaturation level being indicated through the use of an X.

FIG. 27 illustrates normalized pixel values which may be generated fromthe values shown in FIG. 26 in accordance with the method of FIG. 24.

FIG. 28 shows combined image pixel values generated from the input pixelvalues of FIG. 25 using the method of FIG. 24 along with the computationused to produce the illustrated pixel values.

DETAILED DESCRIPTION

FIG. 1A illustrates an exemplary apparatus 100, sometimes referred tohereinafter as a camera device, implemented in accordance with oneexemplary embodiment of the present invention. The camera device 100, insome embodiments, is a portable device, e.g., a cell phone or tabletincluding a camera assembly. In other embodiments, it is fixed devicesuch as a wall mounted camera.

FIG. 1A illustrates the camera device 100 in block diagram form showingthe connections between various elements of the apparatus 100. Theexemplary camera device 100 includes a display device 102, an inputdevice 106, memory 108, a processor 110, a transceiver interface 114,e.g., a cellular interface, a WIFI interface, or a USB interface, an I/Ointerface 112, and a bus 116 which are mounted in a housing representedby the rectangular box touched by the line leading to reference number100. The input device 106 may be, and in some embodiments is, e.g.,keypad, touch screen, or similar device that may be used for inputtinginformation, data and/or instructions. The display device 102 may be,and in some embodiments is, a touch screen, used to display images,video, information regarding the configuration of the camera device,and/or status of data processing being performed on the camera device.In the case where the display device 102 is a touch screen, the displaydevice 102 serves as an additional input device and/or as an alternativeto the separate input device, e.g., buttons, 106. The I/O interface 112couples the display 102 and input device 106 to the bus 116 andinterfaces between the display 102, input device 106 and the otherelements of the camera which can communicate and interact via the bus116. In addition to being coupled to the I/O interface 112, the bus 116is coupled to the memory 108, processor 110, an optional autofocuscontroller 132, a transceiver interface 114, and a plurality of opticalchain modules 130, e.g., N optical chain modules. In some embodiments Nis an integer greater than 2, e.g., 3, 4, 7 or a larger value dependingon the particular embodiment. Images captured by individual opticalchain modules in the plurality of optical chain modules 130 can bestored in memory 108, e.g., as part of the data/information 120 andprocessed by the processor 110, e.g., to generate one or more compositeimages. Multiple captured images and/or composite images may beprocessed to form video, e.g., a series of images corresponding to aperiod of time. Transceiver interface 114 couples the internalcomponents of the camera device 100 to an external network, e.g., theInternet, and/or one or more other devices e.g., memory or stand alonecomputer. Via interface 114 the camera device 100 can and does outputdata, e.g., captured images, generated composite images, and/orgenerated video. The output may be to a network or to another externaldevice for processing, storage and/or to be shared. The captured imagedata, generated composite images and/or video can be provided as inputdata to another device for further processing and/or sent for storage,e.g., in external memory, an external device or in a network.

The transceiver interface 114 of the camera device 100 may be, and insome instances is, coupled to a computer so that image data may beprocessed on the external computer. In some embodiments the externalcomputer has a higher computational processing capability than thecamera device 100 which allows for more computationally complex imageprocessing of the image data outputted to occur on the externalcomputer. The transceiver interface 114 also allows data, informationand instructions to be supplied to the camera device 100 from one ormore networks and/or other external devices such as a computer or memoryfor storage and/or processing on the camera device 100. For example,background images may be supplied to the camera device to be combined bythe camera processor 110 with one or more images captured by the cameradevice 100. Instructions and/or data updates can be loaded onto thecamera via interface 114 and stored in memory 108.

The camera device 100 may include, and in some embodiments does include,an autofocus controller 132 and/or autofocus drive assembly 134. Theautofocus controller 132 is present in at least some autofocusembodiments but would be omitted in fixed focus embodiments. Theautofocus controller 132 controls adjustment of at least one lensposition in the optical chain modules used to achieve a desired, e.g.,user indicated, focus. In the case where individual drive assemblies areincluded in each optical chain module, the autofocus controller 132 maydrive the autofocus drive of various optical chain modules to focus onthe same target. As will be discussed further below, in some embodimentslenses for multiple optical chain modules are mounted on a singleplatter which may be moved allowing all the lenses on the platter to bemoved by adjusting the position of the lens platter. In some suchembodiments the autofocus drive assembly 134 is included as an elementthat is external to the individual optical chain modules with the driveassembly 134 driving the platter including the lenses for multipleoptical chains under control of the autofocus controller 132. While theoptical chain modules will in many embodiments be focused together tofocus on an object at a particular distance from the camera device 100,it is possible for different optical chain modules to be focused todifferent distances and in some embodiments different focus points areintentionally used for different optical chains to increase the postprocessing options which are available.

The processor 110 controls operation of the camera device 100 to controlthe elements of the camera device 100 to implement the steps of themethods described herein. The processor may be a dedicated processorthat is preconfigured to implement the methods. However, in manyembodiments the processor 110 operates under direction of softwaremodules and/or routines stored in the memory 108 which includeinstructions that, when executed, cause the processor to control thecamera device 100 to implement one, more or all of the methods describedherein. Memory 108 includes an assembly of modules 118 wherein one ormore modules include one or more software routines, e.g., machineexecutable instructions, for implementing the image capture and/or imagedata processing methods of the present invention. Individual stepsand/or lines of code in the modules of 118 when executed by theprocessor 110 control the processor 110 to perform steps of the methodof the invention. When executed by processor 110, the data processingmodules 118 cause at least some data to be processed by the processor110 in accordance with the method of the present invention. Theresulting data and information (e.g., captured images of a scene,combined images of a scene, etc.) are stored in data memory 120 forfuture use, additional processing, and/or output, e.g., to displaydevice 102 for display or to another device for transmission, processingand/or display. The memory 108 includes different types of memory forexample, Random Access Memory (RAM) in which the assembly of modules 118and data/information 120 may be, and in some embodiments are stored forfuture use. Read only Memory (ROM) in which the assembly of modules 118may be stored for power failures. Non-volatile memory such as flashmemory for storage of data, information and instructions may also beused to implement memory 108. Memory cards may be added to the device toprovide additional memory for storing data (e.g., images and video)and/or instructions such as programming. Accordingly, memory 108 may beimplemented using any of a wide variety of non-transitory computer ormachine readable mediums which serve as storage devices.

Having described the general components of the camera device 100 withreference to FIG. 1A, various features relating to the plurality ofoptical chain modules 130 will now be discussed with reference to FIGS.1B and 1C which show the camera device 100 from front and sideperspectives, respectively. Dashed line 101 of FIG. 1 B indicates across section line corresponding to the FIG. 1C view.

Box 117 represents a key and indicates that OCM=optical chain module andeach L1 represents an outermost lens in an optical chain module. Box 119represents a key and indicates that S=sensor, F=filter, L=lens, L1represents an outermost lens in an optical chain module, and L2represents an inner lens in an optical chain module.

FIG. 1B shows the front of the camera device 100. Rays of light 131,which is light toward the front of the camera assembly, shown in FIG. 1Cmay enter the lenses located in the front of the camera housing. Fromthe front of camera device 100, the camera device 100 appears as arelatively flat device with the outer rectangle representing the camerahousing and the square towards the center of the camera representing theportion of the front camera body in which the plurality of optical chainmodules 130 is mounted.

FIG. 1C, which shows a side perspective of camera device 100,illustrates three of the seven optical chain modules (OCM 1 121, OCM 7145, OCM 4 133) of the set of optical chain modules 130, display 102 andprocessor 110. OCM 1 121 includes an outer lens L1 103, a filter 123, aninner lens L2 125, and a sensor 127. OCM 1 121 further includesautofocus drive (AFD) 129 for controlling the position of lens L2 125,and exposure control device (ECD) 131 for controlling sensor 127. TheAFD 129 includes a motor or other drive mechanism which can move thelens (or sensor) to which it is connected. While the AFD 129 is showncoupled, e.g., connected, to the lens L2 125 and thus can move theposition of the lens L2 as part of a focus operation, in otherembodiments the AFD 129 is coupled to the sensor 127 and moves theposition of the sensor 127, e.g., to change the distance between thesensor 127 and the lens 125 as part of a focus operation. OCM 7 145includes an outer lens L1 115, a filter 147, an inner lens L2 149, and asensor 151. OCM 7 145 further includes AFD 153 for controlling theposition of lens L2 149 and ECD 155 for controlling sensor 151.

OCM 4 133 includes an outer lens L1 109, a filter 135, an inner lens L2137, and a sensor 139. The AFD 153 includes a motor or other drivemechanism which can move the lens (or sensor) to which it is connected.While the AFD 153 is shown coupled, e.g., connected, to the lens L2 149and thus can move the position of the lens L2 as part of a focusoperation, in other embodiments the AFD 149 is coupled to the sensor 151and moves the position of the sensor 151, e.g., to change the distancebetween the sensor 151 and the lens 149 as part of a focus operation.

OCM 4 133 further includes AFD 141 for controlling the position of lensL2 137 and ECD 143 for controlling sensor 139. The AFD 141 includes amotor or other drive mechanism which can move the lens (or sensor) towhich it is connected. While the AFD 141 is shown coupled, e.g.,connected, to the lens L2 137 and thus can move the position of the lensL2 as part of a focus operation, in other embodiments the AFD 141 iscoupled to the sensor 139 and moves the position of the sensor 139,e.g., to change the distance between the sensor 139 and the lens 137 aspart of a focus operation.

While only three of the OCMs are shown in FIG. 1C it should beappreciated that the other OCMS of the camera device 100 may, and insome embodiments do, have the same or similar structure. FIG. 1C and theoptical chain modules (OCMs), also sometimes referred to as opticalcamera modules, illustrated therein are illustrative of the generalstructure of OCMs used in various embodiments. However, as will bediscussed in detail below, numerous modifications and particularconfigurations are possible. Many of the particular configurations willbe discussed below with use of reference to the optical camera modulesshown in FIG. 1C. While reference to elements of FIG. 1C may be made, itis to be understood that the OCMs in a particular embodiment will beconfigured as described with regard to the particular embodiment. Thus,for example, the filter may be of a particular color. Similarly, inembodiments where the filter is expressly omitted and described as beingomitted or an element which allows all light to pass, while referencemay be made to the OCMs of FIG. 1C, it should be appreciated that thefilter will be omitted in an embodiment where it is indicated to beomitted or of such a nature that it passes a broad spectrum of light topass if the embodiment is indicated to have a broadband filter. As willbe discussed below, the elements of the different OCMs may, but need notbe, mounted on a common support device, e.g., disc or platter, allowinga set of filters, lenses or sensors of the different optical chains tobe moved as a set. While in the OCMs of FIG. 1C mirrors are not shown,as will be discussed below, in at least some embodiments one or moremirrors are added to the OCMs to all light to be directed, e.g., toincrease the length of the optical path or make for a more convenientinternal component configuration. It should be appreciated that each ofthe OCMS 121, 145, 133, shown in FIG. 1C will have their own opticalaxis which corresponds to the path light entering the particular OCMwill follow as it passes from the lens 103, 115, or 109 at the front ofthe optical chain and passes through the OCM to the corresponding sensor127, 151, 139.

While the processor 110 is not shown being coupled to the AFD, ECD andsensors 127, 151, 139 it is to be appreciated that such connectionsexist and are omitted from FIG. 1C to facilitate the illustration of theconfiguration of the exemplary OCMs.

As should be appreciated the number and arrangement of lens, filtersand/or mirrors can vary depending on the particular embodiment and thearrangement shown in FIG. 1C is intended to be exemplary and tofacilitate an understanding of the invention rather than limiting innature.

The front of the plurality of optical chain modules 130 is visible inFIG. 1B with the outermost lens of each optical chain module appearingas a circle represented using a solid line (OCM 1 L1 103, OCM 2 L1 105,OCM 3 L1 107, OCM 4 L1 109, OCM 5 L1 111, OCM 6 L1 113, OCM 7 L1 115).In the FIG. 1B example, the plurality of optical chain modules 130include seven optical chain modules, OCM 1 121, OCM 2 157, OCM 3 159,OCM 4 133, OCM 5 171, OCM 6 173, OCM 7 145, which include lenses (OCM 1L1 103, OCM 2 L1 105, OCM 3 L1 107, OCM 4 L1 109, OCM 5 L1 111, OCM 6 L1113, OCM 7 L1 115), respectively, represented by the solid circles shownin FIG. 1B. The lenses of the optical chain modules are arranged to forma pattern which is generally circular in the FIG. 1B example when viewedas a unit from the front. While a circular arrangement is preferred insome embodiments, non-circular arrangements are used and preferred inother embodiments. In some embodiments while the overall pattern isgenerally or roughly circular, different distances to the center of thegeneral circle and/or different distances from one lens to another isintentionally used to facilitate generation of a depth map and blockprocessing of images which may include periodic structures such asrepeating patterns without the need to identify edges of the repeatingpattern. Such repeating patterns may be found in a grill or a screen.

Note that the individual outer lenses, in combination, occupy an areathat might otherwise have been occupied by a single large lens. Thus,the overall total light capture area corresponding to the multiplelenses of the plurality of chain modules OCM 1 to OCM 7, also sometimesreferred to as optical camera modules, approximates that of a lenshaving a much larger opening but without requiring a single lens havingthe thickness which would normally be necessitated by the curvature of asingle lens occupying the area which the lenses shown in FIG. 1B occupy.

While gaps are shown between the lens openings of the optical chainmodules OCM 1 to OCM 7, it should be appreciated that the lenses may bemade, and in some embodiments are, made so that they closely fittogether minimizing gaps between the lenses represented by the circlesformed by solid lines. While seven optical chain modules are shown inFIG. 1B, it should be appreciated that other numbers of optical chainmodules are possible.

As will be discussed below, the use of seven optical chain modulesprovides a wide degree of flexibility in terms of the types of filtercombinations and exposure times that can be used for different colorswhile still providing an optical camera module that can be used toprovide an image for purposes of user preview of the image area andselection of a desired focal distance, e.g., by selecting an object inthe preview image which is to be the object where the camera modules areto be focused.

For example, in some embodiments, such as the FIG. 6 embodiment, atleast some of the different optical chain modules include filterscorresponding to a single color thereby allowing capture of a singlecolor at the full resolution of the image sensor, e.g., the sensor doesnot include a Bayer filter. In one embodiment two optical chain modulesare dedicated to capturing red light, two optical chain modules arededicated to capturing green light and two optical chain modules arededicated to capturing blue light. The center optical chain module mayinclude a RGB filter or opening which passes all colors with differentportions of the sensor of the center optical chain module being coveredby different color filters, e.g., a Bayer pattern with the optical chainmodule being used to capture all three colors making it easy to generatecolor preview images without having to process the output of multipleoptical chain modules to generate a preview image.

The use of multiple optical chains such as shown in the FIG. 1A-1Cembodiment has several advantages over the use of a single opticalchain.

Using multiple optical chains allows for noise averaging. For example,given the small sensor size there is a random probability that oneoptical chain may detect a different number, e.g., one or more, photonsthan another optical chain. This may represent noise as opposed toactual human perceivable variations in the image being sensed. Byaveraging the sensed pixel values corresponding to a portion of animage, sensed by different optical chains, the random noise may beaveraged resulting in a more accurate and pleasing representation of animage or scene than if the output of a single optical chain was used.

As should be appreciated, different wavelengths of light will be bent bydifferent amounts by the same lens. This is because the refractive indexof glass (or plastic) which the lens is made of changes with wavelength.Dedication of individual optical chains to a particular color allows forthe lenses for those optical chains to be designed taking intoconsideration the refractive index of the specific range of wavelengthfor that color of light. This can reduce chromatic aberration andsimplify lens design. Having multiple optical chains per color also hasthe advantage of allowing for different exposure times for differentoptical chains corresponding to a different color. Thus, as will bediscussed further below, a greater dynamic range in terms of lightintensity can be covered by having different optical chains usedifferent exposure times and then combining the result to form thecomposite image, e.g., by weighting the pixel values output by thesensors of different optical chains as a function of exposure time whencombing the sensed pixel values to generate a composite pixel value foruse in a composite image. Given the small size of the optical sensors(pixels) the dynamic range, in terms of light sensitivity, is limitedwith the sensors becoming easily saturated under bright conditions. Byusing multiple optical chains corresponding to different exposure timesthe dark areas can be sensed by the sensor corresponding to the longerexposure time while the light areas of a scene can be sensed by theoptical chain with the shorter exposure time without getting saturated.Pixel sensors of the optical chains that become saturated as indicatedby a pixel value indicative of sensor saturation can be ignored, and thepixel value from the other, e.g., less exposed, optical chain can beused without contribution from the saturated pixel sensor of the otheroptical chain. Weighting and combining of non-saturated pixel values asa function of exposure time is used in some embodiments. By combiningthe output of sensors with different exposure times a greater dynamicrange can be covered than would be possible using a single sensor andexposure time.

FIG. 1C is a cross section perspective of the camera device 100 shown inFIGS. 1A and 1B. Dashed line 101 in FIG. 1B shows the location withinthe camera device to which the cross section of FIG. 1C corresponds.From the side cross section, the components of the first, seventh andfourth optical chains are visible.

As illustrated in FIG. 1C despite including multiple optical chains thecamera device 100 can be implemented as a relatively thin device, e.g.,a device less than 2, 3 or 4 centimeters in thickness in at least someembodiments. Thicker devices are also possible, for example devices withtelephoto lenses and are within the scope of the invention, but thethinner versions are particularly well suited for cell phones and/ortablet implementations.

As illustrated in the FIG. 1C diagram, the display device 102 may beplaced behind the plurality of optical chain modules 130 with theprocessor 110, memory and other components being positioned, at least insome embodiments, above or below the display and/or optical chainmodules 130. As will be discussed below, and as shown in FIG. 1C, eachof the optical chains OCM 1 121, OCM 7 145, OCM 4 133 may, and in someembodiments do, include an outer lens L1, an optional filter F, and asecond optional lens L2 which proceed a sensor S which captures andmeasures the intensity of light which passes through the lens L1, filterF and second lens L2 to reach the sensor S. The filter may be a colorfilter or one of a variety of other types of light filters.

In FIG. 1C, each optical chain module includes an auto focus drive (AFD)also sometimes referred to as an auto focus device which can alter theposition of the second lens L2, e.g., move it forward or back, as partof a focus operation. An exposure control device (ECD) which controlsthe light exposure time of the sensor to which the ECD corresponds, isalso included in each of the OCMs shown in the FIG. 1C embodiment. TheAFD of each optical chain module operates under the control of theautofocus controller 132 which is responsive to user input whichidentifies the focus distance, e.g., by the user highlighting an objectin a preview image to which the focus is to be set. The autofocuscontroller while shown as a separate element of the device 100 can beimplemented as a module stored in memory and executed by processor 110.

Note that while supporting a relatively large light capture area andoffering a large amount of flexibility in terms of color filtering andexposure time, the camera device 100 shown in FIG. 1C is relatively thinwith a thickness that is much less, e.g., ⅕th, 1/10th, 1/20th or evenless than the overall side to side length or even top to bottom lengthof the camera device visible in FIG. 1B.

FIG. 1D illustrates a plurality of optical chain modules 160 that can beused in an exemplary device implemented in accordance with theinvention. The optical chain modules (OCMs) shown in FIG. 1D areillustrative of the general structure of OCMs used in variousembodiments. However, as will be discussed in detail below, numerousmodifications and particular configurations are possible. Many of theparticular configurations will be discussed below with use of referenceto the optical camera modules shown in FIG. 1D to support the particularexemplary embodiments. While reference to elements of FIG. 1D may andwill be made with regard to particular embodiments, it is to beunderstood that the OCMs in a particular embodiment will be configuredas described with regard to the particular embodiment. Thus, forexample, in a particular embodiment one or of the OCMS may use filtersof a particular color or may even omit the filter 164, 164′. 164″ or164″. Similarly, in embodiments where the filter is expressly omittedand described as being omitted or an element which allows all light topass, while reference may be made to the OCMs of FIG. 1D, it should beappreciated that the filter will be omitted in such an embodiment whereit is expressly indicated to be omitted or of such a nature that itpasses a broad spectrum of light to pass if the embodiment is indicatedto have a broadband filter. As will be discussed below, the elements ofthe different OCMs may, but need not be, mounted on a common supportdevice, e.g., disc or platter, allowing a set of filters, lenses orsensors of the different optical chains to be moved as a set. While inthe OCMs of FIG. 1D mirrors are not shown, as will be discussed below,in at least some embodiments one or more mirrors are added to the OCMsto all light to be directed, e.g., to increase the length of the opticalpath or make for a more convenient internal component configuration. Itshould be appreciated that each of the OCMS 164, 164′, 164″. 164′, shownin FIG. 1C will have their own optical axis which corresponds to thepath light entering the particular OCM will follow as it passes from thelens 162, 162′. 162″, 162′ at the front of the optical chain and passesthrough the OCM to the corresponding sensor 168, 168′, 168″, 168′″.

The plurality of optical chain modules 160 includes N exemplary opticalchain modules as illustrated in FIG. 1D where N may be any number butusually a number greater than one, and in many cases greater than 2, 6or even a larger number. The plurality of optical chain modules 160includes a first optical chain module (OCM) 161, a second optical chainmodule 161′, a third optical chain module 161″, . . . , and Nth opticalchain module 161′″.

Each optical chain module illustrated in FIG. 1D includes many or all ofthe same elements shown in each optical chain illustrated in FIG. 1Csuch as, e.g., optical chain module 121. The first exemplary OCM 161includes an outer lens 162, a filter 164, an inner lens 166, a sensor168, an auto focus drive (AFD) 169 and an exposure control device (ECD)170. Each of the other optical chain modules include similar elements asdescribed above with regard to the first OCM 160, with the like elementsin each of the other optical chain modules being identified using aprime (′), double prime (″), or triple prime (′″). For example, theexemplary second OCM 161′ includes an outer lens 162′, a filter 164′, aninner lens 166′, a sensor 168′, an auto focus drive (AFD) 169′ and anexposure control device (ECD) 170′, the exemplary third OCM 161″includes an outer lens 162″, a filter 164″, an inner lens 166″, a sensor168″, an auto focus drive (AFD) 169″ and an exposure control device(ECD) 170″ and so on. Similarly, the Nth OCM 161′ includes an outer lens162′, a filter 164′″, an inner lens 166′″, a sensor 168′, an auto focusdrive (AFD) 169′ and an exposure control device (ECD) 170′. Theoperation and functionality of each of the OCMs and their elements isthe same as or similar the functionality of optical chain modulesdiscussed earlier with respect to FIG. 1C and thus will not be repeated.Note that two versions of the AFD 169, 169′, 169″ or 169′″ are shown ineach optical chain module with the AFD connected to a lens being shownusing solid lines and an alternative AFD shown using dashed lines beingconnected to the sensor 168, 168′, 168″ or 168′″. The AFD shown withdashed lines adjusts the position of the sensor 168. 168′, 168″ or 168′″to which it is connected as part of an autofocus operation, e.g., movingthe sensor forward or backward to alter distance between the sensor anda lens. The AFD shown in solid lines is used in systems where a lensrather than a sensor is moved as part of an AFD operation. In someembodiments the AFD controls the position of a lens and/or sensor inwhich case the AFD module is connected to both a lens support mechanismor lens and the sensor.

The plurality of optical chain modules 160 of FIG. 1D can be used as,e.g., the plurality of optical modules 130 of the exemplary device 100or any other device implemented in accordance with the invention. Thenumber and particular configuration of optical chains in the step ofoptical chains 160 maybe as per various embodiments which will bedescribed in the following detailed description. Accordingly, while aparticular embodiment may be described in one more subsequent portionsof this application, it is to be understood that the optical chains 160may be used in such embodiments with the particular configuration offilters, lens, and element supports being as described with respect tothe particular exemplary embodiment being discussed.

FIG. 2 illustrates a camera device 200 implemented in accordance withthe invention. The FIG. 2 camera device 200 includes many or all of thesame elements shown in the device 100 of FIGS. 1A-1C. Exemplary cameradevice 200 includes a plurality of optical chain modules (OCM 1 205, OCM2 207, . . . , OCM N 209, a processor 211, memory 213 and a display 215,coupled together. OCM 1 205 includes outer lens L1 251, filter 253,inner lens L2 255, sensor 1 257, AFD 259 and ECD 261. In someembodiments, processor 211 of camera device 200 of FIG. 2 is the same asprocessor 110 of device 100 of FIG. 1A, memory 213 of device 200 of FIG.2 is the same as memory 108 of device 100 of FIG. 1A, and display 215 ofdevice 200 of FIG. 2 is the same as display 102 of device 100 of FIG.1A.

OCM 2 207 includes outer lens L1 263, filter 265, inner lens L2 267,sensor 2 269, AFD 271 and ECD 273. OCM N 209 includes outer lens L1 275,filter 277, inner lens L2 279, sensor N 281, AFD 283 and ECD 285. Box217, which represents a key, indicates that ECD=exposoure control deviceand AFD=auto focus drive.

In the FIG. 2 embodiment the optical chain modules (optical chain module1 205, optical chain module 2 207, . . . , optical chain module N 209)are shown as independent assemblies with the autofocus drive of eachmodule being a separate AFD element (AFD 259, AFD 271, AFD 283),respectively.

In FIG. 2, the structural relationship between the various lenses andfilters which precede the sensor in each optical chain module can beseen more clearly. While three elements, e.g. two lenses (see columns201 and 203 corresponding to L1 and L2, respectively) and the filter(corresponding to column 202) are shown in FIG. 2 before each sensor, itshould be appreciated that a much larger combination of lenses and/orfilters may precede the sensor of one or more optical chain modules withanywhere from 2-10 elements being common and an even larger number ofelements being used in some embodiments, e.g., high end embodimentsand/or embodiments supporting a large number of filter and/or lensoptions.

In some but not all embodiments, optical chain modules are mounted inthe camera device to extend from the front of the camera device towardsthe back, e.g., with multiple optical chain modules being arranged inparallel. Filters and/or lenses corresponding to different optical chainmodules may, and in some embodiments are, arranged in planes extendingperpendicular to the front to back direction of the camera device fromthe bottom of the camera device towards the top of the camera device.While such a mounting arrangement is used in some embodiments, otherarrangements where the optical chain modules are arranged at differentangles to one another and/or the camera body are possible.

Note that the lenses/filters are arranged in planes or columns in thevertical dimension of the camera device 200 to which reference numbers201, 202, 203 correspond. The fact that the lenses/filters are alignedalong vertical planes allows for a manufacturing and structuralsimplification that is used in some embodiments. That is, in someembodiments, the lenses and/or filters corresponding to a plane 201,202, 203 are formed or mounted on a platter or plate. The term platterwill be used for discussion purposes but is not intended to be limiting.The platter may take the form of a disc but non-round platters are alsocontemplated and are well suited for some embodiments. In the case ofplastic lenses, the lenses and platter may be molded out of the samematerial in a single molding operation greatly reducing costs ascompared to the need to manufacture and mount separate lenses. As willbe discussed further, platter based embodiments allow for relativelysimple synchronized focus operations in that a platter may be movedfront or back to focus multiple OCMs at the same time. In addition, aswill be explained, platters may be moved or rotated, e.g., along acentral or non-central axis, to change lenses and or filterscorresponding to multiple optical chain modules in a single operation. Asingle platter may include a combination of lenses and/or filtersallowing, e.g., a lens to be replaced with a filter, a filter to bereplaced with a lens, a filter or lens to be replaced with anunobstructed opening. As should be appreciated the platter basedapproach to lens, filter and/or holes allows for a wide range ofpossible combinations and changes to be made by simple movement of oneor more platters. It should also be appreciated that multiple elementsmay be combined and mounted together on a platter. For example, multiplelenses, filters and/or lens-filter combinations can be assembled andmounted to a platter, e.g., one assembly per optical chain module. Theassemblies mounted on the platter for different optical chains may bemoved together, e.g., by rotating the platter, moving the platterhorizontally or vertically or by moving the platter using somecombination of one or more such movements.

While platters have been described as being moved to change elements inan optical chain, they can, and in some embodiments are, moved for imagestabilization purposes. For example, a platter having one or more lensesmounted thereon can be moved as part of an image stabilizationoperation, e.g., to compensate for camera motion.

While mounting of lenses and filters on platters has been discussed, itshould also be appreciated that the sensors of multiple optical chainscan be mounted on a platter. For example, sensors without color filtersmay be replaced with sensors with color filters, e.g., Bayer patternfilters. In such an embodiment sensors can be swapped or changed whileleaving one or more components of one or more optical chains in place.

Note from a review of FIG. 2 that in some embodiments, e.g., largerfocal length telephoto applications, the elements, e.g., filters/lensescloser to the sensor of the optical chain module, are smaller in sizethan the outer most lenses shown in column 201. As a result of theshrinking size of the lenses/filters, space becomes available betweenthe lenses/filters within the corresponding platter.

FIGS. 3A through 3C provide perspective views of the different planes201, 202, 203 shown in FIG. 2. As shown in FIG. 3A, the outer lenses L1(OCM 1 L1 251, OCM 2 L1 263, OCM 3 L1 264, OCM 4 L1 266, OCM 5 L1 268,OCM 6 L1 270, OCM 7 L1 272) occupy much of the outer circular areacorresponding to the front of the camera modules as previously shown inFIG. 1B. However, as shown in FIG. 3B the filters (OCM 1 F 253, OCM 2 F265, OCM 3 F 274, OCM 4 F 276, OCM 5 F 278, OCM 6 F 280, OCM 7 F 282)corresponding to plane 202 occupy less space than the lenses shown inFIG. 3A while the inner lenses L2 (OCM 1 L2 255, OCM 2 L2 267, OCM 3 L2284, OCM 4 L2 286, OCM 5 L2 288, OCM 6 L2 290, OCM 7 L2 292) shown inFIG. 3C occupy even less space. In some embodiments, where N=7, outerlens L1 275, filter F 277, and inner lens L2 279 of FIG. 2 are the sameas OCM 7 L1 272 of FIG. 3A, OCM 7 F 282 of FIG. 3B and OCM 7 L2 292 ofFIG. 3C, respectively.

The decreasing size of the inner components allow multiple lenses and/orfilters to be incorporated into a platter corresponding to one or moreof the inner planes. Consider for example that an alternative filter F′or hole could be mounted/drilled below or next two each filter F of aplatter corresponding to plan 202 and that by shifting the position orplatter vertically, horizontally or a combination of horizontally andvertically, the filter F can be easily and simply replaced with anotherfilter or hole. Similarly the lenses L2 may be replaced by alternativelenses L2′ by shifting a platter of lenses corresponding to plane 203.In some embodiments, the platter may also be rotated to support changes.The rotation may be an off center rotation and/or may be performed incombination with one or more other platter position changes.

A camera device 60 which includes platters of lenses and/or filters (61,62, 63) is shown in FIG. 4. Camera device 60 includes a plurality ofoptical chain modules (optical chain module 1 69, optical chain module 270, . . . , optical chain module N 71), processor 72, memory 73, anddisplay 74 coupled together via bus 75. In some embodiments, processor72, memory 73, display 74, and autofocus controller 76 of device 60 ofFIG. 4 are the same as processor 110, memory 108, display 102, andautofocus controller 132 of device 100 of FIG. 1A.

Element 61 represents a platter of outer lenses L1 with 3 of the lenses(77, 81, 86) being shown as in the FIG. 1C example. Additional lensesmay be, and often are, included on the platter 61 in addition to theones shown. For example, in a seven optical chain module embodiment suchas shown in FIG. 1, platter 61 would include seven outer lenses. Notethat the thickness of the platter 61 need not exceed the maximumthicknesses of the lenses and from a side perspective is much thinnerthan if a single lens having a similar curvature to that of theindividual lenses L1, but with the single lens being larger, occupiedthe same area as all the 7 lenses on the platter 61. Platter 62 includesthe filters F, which include the three filters (77, 82, 87) whileplatter 63 includes the inner lenses L2, which include the three lenses(78, 83, 88). As can be appreciated the camera device 60 is the same asor similar to the camera device of FIG. 1C and FIG. 2 but with thelenses and filters being mounted on platters which may be moved betweenthe front and back of the camera to support autofocus or horizontallyand/or vertically to support lens/filter changes.

Auto focus drive 66 is used to move platter 63 forward or backward aspart of a focus operation, e.g., under control of the autofocuscontroller 76 which may be, and often is, included in the camera device60. A filter shift drive (FSD) 65 is included in embodiments whereshifting of the platter 62 is supported as part of a filter changeoperation. The FSD 65 is responsive to the processor 72 which operatesin response to user selection of a particular mode of operation and/oran automatically selected mode of operation and can move the platter 62vertically, horizontally or in some combination of vertical andhorizontal motion to implement a filter change operation. The FSD 62 maybe implemented with a motor and mechanical linkage to the platter 62. Insome embodiments, the platter 62 may also be rotated to support changes.The rotation may be an off center rotation and/or may be performed incombination with one or more other platter position changes.

A lens shift drive (LSD) 67 is included in embodiments where shifting ofthe platter 63 is supported as part of a filter change operation. TheLSD 67 is responsive to the processor 72 which operates in response touser selection of a particular mode of operation and/or an automaticallyselected mode of operation and can move the platter 63 vertically,horizontally or in some combination of vertical and horizontal motion toimplement a lens change operation. The LSD 67 may be implemented with amotor and mechanical linkage to the platter 63. In some embodiments, theplatter 63 may also be rotated to support changes. The rotation may bean off center rotation and/or may be performed in combination with oneor more other platter position changes.

FIG. 5A illustrates various exemplary platters that can, and in someembodiments are, used as the filter platter and/or inner lens platter inthe camera device 60 of FIG. 4. In the FIG. 5A example N is three (3)but other values of N are possible depending on the embodiment. FIG. 5Bshows the exemplary lens platter 62′ of FIG. 5A when viewed from theside, drawing 6299, and from the front, drawing 6298.

Platter 62 represents a platter with a single set of filters F1,1 6202corresponding to OCM1, F1,2 6204 corresponding to OCM 2 and F1,3 6206corresponding to OCM 3.

Platter 62′ represents an alternative platter that can, and in someembodiments is, used in place of platter 62. NF is use to represent ahole or No Filter (NF) area of the platter 62′. As should be appreciatedby simply shifting platter 62′ vertically the filters F1 (F1,1 6202,F1,2 6204, F1, 3 6206) can be replaced by holes (NF 6208, NF 6210, NF6212), respectively, thereby removing the color or other types offilters previously included in the optical chain modules.

Platter 62″ of FIG. 5A represents a platter which includes alternativefilters F1′ (F1′, 1 6214, F1′, 2 6216, F1′ 3 6206) which can be switchedfor the filters F1 (F1, 1 6202, F1,2 6204, F1,3 6206), respectively, bymoving the platter 62″ vertically. Thus platter 62″ is used to show howfilters can be switched for other filters by simple movement of aplatter while platter 62′ shows how filters can be removed from theoptical paths included in a plurality of optical chain modules byshifting of the platter on which a set of filters are mounted.

With regard to drawing 6298 of FIG. 5B, as should be appreciated bysimply shifting platter 62′ vertically the filters F1 (F1,1 6202, F1,26204, F1, 3 6206, F1,4 6220, F1, 5 6224, F1, 6 6228, F1, 7 6232) can bereplaced by holes (NF 6208, NF 6210, NF 6212, NF 6222, NF 6226, NF 6230,NF 6234), respectively, thereby removing the color or other types offilters previously included in the optical chain modules.

Lens platter 63 shows a platter of inner lenses L2 (L2,1 6302, L2,26304, L2,3 6306) corresponding to first, second and third optical cameramodules. Lens platter 63′ is an alternative platter which shows howalternative lenses L2′ (L2′,1 6308, L2′,2 6310, L2′,3 6312) can beincluded on a lens platter and easily swapped for the lenses L2 (L2,16302, L2,2 6304, L2,3 6306), respectively, by simple movement of theplatter 63′ vertically or horizontally. Lens platter 63″ is used to showthat a lens platter may include holes (6314, 6316, 6318) as analternative to alternative lenses. Any of lens platters 63, 63′ or 63″could be used in the camera device 60 shown in FIG. 4. While two lenssets are included in platter 63′, multiple lens and/or holecombinations, e.g., 2, 3 or more, may be included in a single platter.Similarly a large number of alternative filter, hole alternatives may besupported in a single filter platter. A platter can also havecombinations of lenses, filters and holes and filters could be swappedfor lenses or holes.

As should be appreciated given the larger number of lens/filtercombinations that can be supported through the use of platters, a singlecamera device including a number of optical chain modules may support alarge number of alternative modes of operation.

It should be appreciated that the exposure control of various opticalchain modules may be varied along with the filters and/or lenses used atany given point in time allowing for a wide degree of flexibility andcontrol over the images captured at any given point in time.

FIGS. 6A, 6B and 6C correspond to one particular filter lens combinationused in some embodiments.

FIG. 6A shows the use of 7 optical chain modules at plane 201 (the outerlens plane corresponding to lenses L1) as viewed from the front of thecamera device. FIG. 6A shows optical chain module L1 lenses (OCM 1 L16402, OCM 2 L1 6404, OCM 3 L1 6406, OCM 4 L1 6408, OCM 5 L1 6410, OCM 6L1 6412, OCM 7 L1 6414). FIG. 6C shows the inner lens plane 203. FIG. 6Cshows optical chain module L2 lenses (OCM 1 L2 6602, OCM 2 L2 6604, OCM3 L2 6606, OCM 4 L2 6608, OCM 5 L2 6610, OCM 6 L2 6612, OCM 7 L2 6614).The configuration shown in FIGS. 6A and 6C is the same or similar tothat previously discussed with reference to the FIG. 3 embodiment. FIG.6B shows a particular color filter arrangement used in some embodiments.The filter arrangement shown in FIG. 6B may be used at filter plane 202.The filter arrangement shown in FIG. 6B may be used in the set ofoptical chain modules 130 before the sensors, e.g., between the set ofL1 and L2 lenses. However, this position is not required for someembodiments and the user of inner lenses L2 is also not required forsome embodiments.

The filter configuration 6002 of FIG. 6B includes single color filtersin each of a plurality of optical chain modules, e.g., the six outeroptical chain modules (OCM1 to OCM6). Multiple optical chain modules arededicated to each of the three colors, red (R), green (G) and blue (B).The optical chain modules (OCM1, OCM4) with the red filter (RF), (OCM 1RF 6502, OCM 4 RF 6508) pass and sense red light. The optical chainmodules (OCM 2, OCM 5) with the green filter (GF), OCM 2 GF 6504, OCM 5GF 6510, pass and sense green light. The optical chain modules (OCM 3,OCM 6) with the blue filter (BF), OCM 3 BF 6506, OCM 6 BF 6512, pass andsense blue light. In various embodiments, there is a single color filterper lens for the outer lenses, e.g., a single color filter correspondingto each of OCM 1-OCM 6. In some such embodiments, there are multipleOCMs per single color, e.g., 2 OCMs for each of Red, Green, and Blue.

By using optical chain modules dedicated to a single color, the opticalchains can be optimized for the spectral range corresponding to theparticular color to which the chain corresponds. In addition postcapture color compensation can be simplified since each of the six outeroptical modules capture a single known color. In addition, noise can beaveraged between the sensor corresponding to the same color and/ordifferent exposure times can be used for the different OCMscorresponding to an individual color extending the dynamic range of thesensors to cover a range wider than could be captured by a singlesensor. In addition different exposure times may be used for differentcolors to take into consideration particular color biased lightingconditions and/or facilitate the implementation of particular coloreffects that may be desired. Notably the individual colors are capturedat a pixel result in a resolution equal to that of the sensor as opposedto the case where different portions of a single sensor are used tocapture different colors, e.g., with each color R, G, B being capturedat a resolution ⅓ that of the pixel resolution of the image sensor beingused in an optical chain module.

In some embodiments, there is a RGB Multicolor Filter, OCM 7 RGBF 6514,corresponding to OCM 7.

In some embodiments, OCM 7 filter 6514 is a RGB filter, e.g., a Bayerfilter.

In some embodiments, an opening which allows all colors to pass is usedin place of OCM 7 RGB filter 6514, but the sensor area corresponding toOCM 7 includes R, G, and B filters corresponding to different sensorarea portions. In some embodiments, OCM 7 is used for preview.

In various embodiments, the sensors for OCM 1 through OCM 6 have nofilters.

While in some embodiments a composite image is generated and displayedas a preview image, in some embodiments to reduce processing time and/orthe time required to display a composite image which may be delayed bythe time required to combine multiple images, an image captured by asingle sensor is displayed as the preview image on the display of thecamera device. The multi-colored filter incorporated into the sensor,e.g., Bayer filter, of OCM 7 allows a color image to be captured by asingle lens and used as the preview image. While the image may be oflower quality than that which can be generated by creating a compositeof the multiple OCMs given the small display size the difference inimage quality between the preview image generated from OCM 7 and that ofa composite image may not be sufficient to justify the processing,power, and/or time required to generate a composite image for previewpurpose. Accordingly, the FIG. 6B filter arrangement provides a greatdeal of flexibility while being able to support a wide variety ofexposure and other image capture related features.

Box 6003 of FIG. 6B identifies that GF=green filter, BF=blue filter,RF=red filter, RGBF=Red, Green, Blue Filter.

The ability to use different exposure times with different optical chainmodules is illustrated further with regard to a camera embodiment whichwill now be discussed with regard to FIGS. 7A, 7B and 7C. The lensconfigurations of FIGS. 7A and 7C are similar to that shown in FIGS. 6Aand 6C. FIG. 7A shows optical chain module L1 lenses (OCM 1 L1 7402, OCM2 L1 7404, OCM 3 L1 7406, OCM 4 L1 7408, OCM 5 L1 7410, OCM 6 L1 7412,OCM 7 L1 7414), which may be located a plane 201. FIG. 6C shows opticalchain module L2 lenses (OCM 1 L2 7602, OCM 2 L2 7604, OCM 3 L2 7606, OCM4 L2 7608, OCM 5 L2 7610, OCM 6 L2 7612, OCM 7 L2 7614), which may belocated at plane 203. The filter arrangement shown in drawing 7002 ofFIG. 7B is also the same or similar to that shown in FIG. 6B but in theFIG. 7B example exposure time is also included. While the exposure iscontrolled by use of the exposure control device in some embodiments theconcept can be understood from FIG. 7B. In FIG. 7B SE is used toindicate short exposure, LE is used to indicate long exposure, and ME isused to indicate medium exposure, as indicated by box 7003. Element 7502indicates that OCM 1 uses a red filter and is controlled for a mediumexposure. Element 7504 indicates that OCM 2 uses a green filter and iscontrolled for a short exposure. Element 7506 indicates that OCM 3 usesa blue filter and is controlled for a short exposure. Element 7508indicates that OCM 4 uses a red filter and is controlled for a longexposure. Element 7510 indicates that OCM 5 uses a green filter and iscontrolled for a long exposure. Element 7512 indicates that OCM 6 uses ablue filter and is controlled for a long exposure. Element 7514indicates that OCM 7 uses a RGB filter, e.g., a Bayer filter, and iscontrolled for medium exposure.

For the outer OCMs, OCM 1 through OCM 6, there is a single color filterper OCM, and multiple OCMs per color. In various embodiments, the centerOCM, OCM 7, is used for preview.

In some embodiments, filters, corresponding to OCM 1 through OCM 7, areincluded at plane 202. In some embodiments, the filters corresponding toOCM 1 through OCM 6 are included at plane 202; there is an opening atplane 2 corresponding to OCM 7, which allows all the colors to pass; andthe sensor area corresponding to OCM 7 includes R, G, and B filterscorresponding to different sensor area portions, e.g., the sensor forOCM 7 includes an RGB Bayer filter. In some embodiments, the sensors forOCM 1 through OCM 6 have no filters.

The preview image is generated using the medium exposure optical chainmodule while the two different optical chain modules corresponding to agiven color use different exposures. In this way the short exposure timecan be used to reliably capture information corresponding to light(e.g., bright) portions of an image while the long exposure opticalchain module can be used to capture information corresponding to thedarker portions of an image. As discussed above, the sensed pixel valuesfrom the two optical chains can be processed to exclude values generatedby saturated sensors and to combine pixel values corresponding to thesame image area in a manner weighted according to the exposure durationfor pixel value within the acceptable operating range of the opticalchain module's sensors.

While different durations can and often are achieved by controllingsensor exposure times, different filters in different optical chainmodules may, and are, used to achieve different light exposures in someembodiments.

FIG. 8, illustrates an optical chain arrangement used in one panoramiccamera device 8000 in which multiple optical chains and different lensangles are used to capture images that are well suited for combininginto a panoramic image. A1 represents a first non-zero angle, Srepresents a straight or 0 degree angle, and A2 represents a secondnon-zero angle. In one embodiment A1 causes the corresponding camerachain module to capture images to the right of the camera, S causes thecorresponding camera chain module to capture images straight ahead ofthe camera, and A2 causes the corresponding camera chain module tocapture images to the left the camera, from the perspective of the userbehind the camera. In addition to captured images left and right of thecamera it should be appreciated that the optical chain modules capturesome image portion which is also captured by the adjacent optical chainmodule. Thus, the OCMs in columns 803, 805 and 807 capture differentscenes which, while overlapping, can be stitched together to provide anultra wide angle panoramic image. The OCMs in each of rows 811, 813,815, 817, 819, 821, 823 capture different versions of the same scene.

The panoramic camera device 8000 includes multiple optical chain modulescorresponding to each of the left, right and center views. Twenty oneoptical chain modules (seven sets of three) are shown allowing for twooptical chain modules per color (R, G, B) plus a seventh multi-color (R,G, B) optical chain module which can be used to support a preview modeof operation. The multi-color optical chain module may include a sensorwith a multicolor filter, e.g., a Bayer pattern filter, allowing thesingle sensor to capture the multiple colors using different portions ofthe sensor. While the panoramic configuration shown in FIG. 8 isdifferent from that of the non-panoramic camera embodiments previouslydiscussed the exposure control and separate color capture benefitsremain the same as those discussed with regard to the other embodiments.

While FIG. 8 illustrates a particular panoramic embodiment, it should beappreciated that embodiments such as those shown in FIGS. 3 and 4 can,and in sometimes are, used to support taking of panoramic pictures. Inone such embodiment a prism or angled lens is inserted into one or moreoptical chain modules, e.g., by rotation, vertical movement, horizontalmovement and/or a combination of vertical and horizontal movement of aplatter upon which the prism or lens is mounted. The prisms or changesin lens angles change the scene area perceived by one or more opticalchain modules allowing the different optical chain modules to capturedifferent views of a scene which can, and in some embodiments are, usedto generate a panoramic image, e.g., picture. Thus, camera modules usedto capture images corresponding to the same scene which are thencombined to generate a combined image can also be used at a differenttime to capture images corresponding to different views and/or sceneswhich can then be subsequently combined to form a panoramic image, e.g.,photograph.

Accordingly, it should be appreciated that ultra wide angle panoramicimages can be generated using multiple optical chain modules of the typepreviously discussed thereby providing panoramic cameras many of thebenefits of large lens without the need for the camera depth, weight andother disadvantages associated with large lenses.

It should be appreciated that because camera chain modules are separatedfrom one another the multi-optical chain module embodiments of thepresent invention are well suited for stereoscopic image generation andfor generating image depth maps. Accordingly the camera devices of thepresent invention support a wide range of applications and modes ofoperation and provide significant amounts of image data which can beused to support a wide range of post capture image processingoperations.

Having described apparatus and various embodiments, various methodswhich are supported and used in some embodiments will now be discussedwith regard to various flow charts that are included in the presentapplication.

Method 300 of FIG. 9 illustrates one exemplary method of producing atleast one image of a first scene area in accordance with the presentinvention. The processing steps of the method 300 of FIG. 9 will now beexplained in view of the camera device 100 of FIG. 1A.

The method 300 of FIG. 9 starts at start step 302 with the start of thesteps of the method being implemented, e.g., on processor 110. Operationproceeds from start step 302 to step 304. In step 304, user input isreceived to control the capture of at least one image of the first scenearea. The user input is received via input device 106 which may be, andin some embodiments is, a button or touch sensitive screen. In optionalsub-step 306, the user input may, and in some embodiments does, indicatea portion of the first scene area that is to be focused, e.g., in animage to be captured or a combined image to be generated from two ormore captured images. From step 304 processing proceeds to step 308.

In step 308, a plurality of three or more optical chain modules (OCMs),e.g., optical chain modules 130 of FIG. 1A, are operated in parallel tocapture images of the first scene area, said images including at least afirst image of said first scene area, a second image of said first scenearea, and a third image of said first scene area. In some embodimentseach one of the first, second and third optical chain modules captures acorresponding one of the first, second and third image respectively. Insome embodiments, operating a plurality of three or more optical chainmodules in parallel to capture images of the first scene area, saidimages including at least a first image of said first scene area, asecond image of said first scene area, and a third image of said firstscene area includes sub-processing steps 310, 312, and 314.

In sub-step 310 a first optical chain module is operated to capture afirst image 316 of the first scene area. In most, but not all,embodiments, on capture of the first image 316, the image data and otherdata such as camera device configuration information associated with thefirst image is stored in the data/information 120 portion of memory 108for later processing, output or display. In parallel with the processingof sub-step 310 processing of sub-steps 312 and 314 also occur. Insub-step 312 a second optical chain module is operated to capture asecond image 318 of the first scene area. In most, but not all,embodiments on capture of the second image 318, the image data and otherdata such as camera device configuration information associated with thesecond image is stored in the data/information 120 portion of memory 108for later processing, output or display. In sub-step 314 a third opticalchain module is operated to capture a third image 320 of the first scenearea. In most, but not all, embodiments on capture of the third image320, the image data and other data such as camera device configurationinformation associated with the third image is stored in thedata/information 120 portion of memory 108 for later processing, outputor display. Processing then proceeds from step 308 to step 322.

In some embodiments, each optical chain module of the plurality ofoptical chain modules includes a lens and the lenses of the plurality ofthe optical chain modules are arranged along a circle. For example, whenthere are three optical chain modules, i.e., a first optical chainmodule, a second optical chain module, and a third optical chain module,the first optical chain module includes a first lens, the second opticalchain module includes a second lens, and the third optical chain moduleincludes a third lens. The first, second and third lenses are arrangeduniformly along a circle, e.g. on the vertices of an equilateraltriangle. In some embodiments the camera device 100 includes a fourthoptical chain module including a fourth lens, said fourth lens beingpositioned in the center of the circle. Each of the first, second, thirdand fourth lens may be, and in some embodiments of the present inventionare, the outer lens of each of their respective optical chain modulesand are all positioned in the same plane. More generally, in someembodiments of the present invention, there are a plurality of N opticalchain modules each including a lens. N−1 lenses of the plurality ofoptical chain modules are arranged along a circle with Nth lens beingpositioned in the center of the circle. FIG. 1B illustrates and exampleof a camera device 100 with seven optical chain modules which include 7outer lenses shown as circles, i.e., OCM1, OCM2, OCM3, OCM4, OCM5, OCM6,and OCM7. The outer lens of optical chain modules OCM 1, OCM2, OCM3,OCM4, OCM5, and OCM6 are arranged along a circle and the outer lens ofoptical chain module OCM7 is positioned in the center of the circle.

In some embodiments of the present invention, the first optical chainmodule includes in addition to the first lens an image sensor referredto as a first image sensor. In some embodiments of the presentinvention, the second optical chain module includes an image sensorreferred to as a second image sensor. In some embodiments of the presentinvention, the third optical chain includes an image sensor referred toas a third image sensor. In some embodiments of the present inventionthe plurality of lenses of the plurality of optical chain modules aremounted in a cell phone housing with the plurality of lenses oriented inthe same direction and in the same plane of the housing. For example inthe case of three optical chain modules, in some embodiments of thepresent invention, the first, second and third lenses of the first,second, and third optical chain modules respectively are mounted in acell phone housing and are oriented in the same direction and in thesame plane of the housing.

In step 322, said first, second, and third images are processed byprocessor 110 to generate a first combined image 326 of said first scenearea. In some embodiments, including those embodiments of the presentinvention in which user input is received indicating a portion of thefirst scene area to be focused in the combined image, step 322 may, andin some embodiments does, include sub-step 324 wherein pixel positionson at least one of said first, second, and third images is shifted priorto generating said first combined image to align the portion of thefirst scene to be focused. Processing then proceeds to step 328 wherethe generated combined image is stored in data/information 120 of memory108, e.g., for potential later display, output from the camera device,and/or additional processing and/or displayed on display 102 of cameradevice 100.

In some embodiments, processing step 322 and/or sub-step 324 areperformed on an external device such as a computer. In such cases, thefirst, second and third images are outputted from the camera device 100via transceiver 114 to the external computer for processing to generatethe first combined image 326. The first combined image may then bestored in memory associated with the external device and/or displayed ona display associated with the external computer. In some embodiments ofthe present invention, the first combined image of the first scene areaincludes the same or fewer pixel values than either of said first,second or third images.

From step 328 processing proceeds to step 304 where processing continuesand the method is repeated.

In some embodiments of the present invention, the size of the diameterof the first, second and third lens of the first, second, and thirdoptical chain modules respectively are the same and the sensors of thefirst, second and third optical chain modules have the same number ofpixels. In other embodiments of the present invention, one or moreoptical chain modules may, and in some embodiments do, have lenses withdifferent diameter sizes and/or sensors with different numbers ofpixels. In some embodiments of the present invention, the first, secondand third lenses of the first, second and third optical chain modulesrespectively, are less than 2 cm in diameter and each of the first,second and third image sensors of the first, second and third opticalchain modules support at least 8 Mpixels. In some embodiments of thepresent invention, the first and second lenses are each less than 2 cmin diameter and each of the first and second image sensors support atleast 5 Mpixels. However in many embodiments the image sensors support 8Mpixels or even more and in some embodiments the lenses are larger than2 cm. Various combinations of lens and sensors may be used with avariety of lens sizes being used for different optical chains in someembodiments. In addition different optical chains may use lenses withdifferent shapes, e.g., while the lens may be a spherical lens theperimeter of the lens may be cut into one of a variety of shapes. In oneembodiment, lenses of different optical chain modules are shaped andarranged to minimize gaps between lenses. Such an approach can have theadvantage of resulting in a smoother blur with regard to portions ofcaptured images which are out of focus when combining images captured bydifferent optical chain modules and result in an overall image whichmore closely approximates what might be expected had a single large lensbeen used to capture the scene shown in the combined image.

In accordance with some aspects of the present invention, the diametersize and arrangement of the lenses of the plurality of optical modulesmay and do vary. Similarly the number of pixels supported by the sensorsof each of the plurality of optical modules may also vary for exampledepending on the desired resolution of the optical chain module.

In some embodiments, different shifts are used for different portions ofthe scene to create a single composite image. In some embodiments, thegenerated combined image is a panoramic image.

In various embodiments, the optical chain modules are independentlyfocused to the same focal distance. In some embodiments, the opticalchain modules are focused together. In some such embodiments, theoptical chain modules are focused together by moving a platter on whichlenses corresponding to different optical chains are mounted.

Method 400 of FIG. 10 illustrates an embodiment of a method of producingat least one image of a first scene area in accordance with the presentinvention. The method 400 achieves enhanced sensor dynamic range bycombining images captured through the operation of two or more opticalchain modules using different exposure times. The processing steps ofthe method 400 of FIG. 10 will now be explained in view of the cameradevice 100 of FIG. 1A. For ease of explanation of the method 400, itwill be assumed that the plurality of optical chain module 130 of cameradevice 100 of FIG. 1A includes two optical chain modules and in someembodiments an optional third optical chain module which will bereferred to as a first, second and third optical chain modulerespectively.

The method 400 of FIG. 10 starts at start step 402 with the start of thesteps of the method being implemented, e.g., on processor 110. Operationproceeds from start step 402 to step 404. In step 404, one of aplurality of optical chain modules of the camera device is operated tocapture an image which will be referred to herein as a fourth image ofthe first scene. For example, one of said first, second or optionalthird optical chain modules may be, and in some embodiments is, operatedto capture the fourth image. This fourth image is captured prior tocapturing the first, second or third images which will be discussed inconnection with step 410 below.

Processing then proceeds to step 406 where the fourth image is displayedon the display 102 of the camera device 100. By displaying the fourthimage on the display of the camera device 100 a user can aim the cameradevice and target the first scene area for which the user wants tocapture an image. In some embodiments, the fourth image is also storedin data/information 120 of memory 108. Processing then proceeds fromstep 406 to step 408.

In step 408, user input is received to control the capture of an imageof the first scene area. The user input is received via input device 106which may be, and in some embodiments is, a button or touch sensitivescreen. For example, the user may touch a portion of the touch sensitivescreen on which the fourth image is shown to focus the camera on aportion of the scene for which an image is to be captured. From step 408processing proceeds to step 410 where the plurality of optical chainmodules 130 are operated in parallel to capture images of the firstscene area.

Step 410 includes sub-steps 412, 414, and optional sub-step 416. Insub-step 412, a first optical chain module is operated to capture afirst image 418 of the first scene area using a first exposure time. Insub-step 414, a second optical chain module is operated to capture asecond image 420 of the first scene area using a second exposure time,at least said first and said second exposure times being of differentduration but overlapping in time. In some embodiments, an optionalsub-step 416 is performed wherein a third optical chain module isoperated to capture a third image 422 of the first scene area using athird exposure time. In some embodiments, the third exposure time isdifferent than the first and second exposure times. Additional opticalchain modules may be, and in some embodiments are, used to captureadditional images of the first scene area with the additional opticalchain modules using the same or different exposure times as the first,second or third exposure times so as to obtain additional image data forthe first scene area. Sub-steps 412, 414, and optional sub-step 416 areperformed in parallel so that multiple images of the first scene arecaptured in parallel with different exposure times. The first, secondand optional third captured images may be, and in some embodiments are,stored in data/information 120 of memory section 108 to be available forlater use such as for example in later steps of the method forgenerating a combined image of the first scene area, or for display oroutputting of images.

In some embodiments, in step 404 the operation of one of the first,second and third optical chain modules to capture the fourth image ofthe first scene area uses a fourth exposure time different from saidfirst, second and third exposure times. Once again step 404 occurs priorto the step 410 as the fourth image is displayed on the display 102 sothe user can utilize the displayed image to target the scene area to becaptured by the first, second and optional third images.

Operation of the method proceeds from step 410 to step 424. In step 424the captured images, that is the first and second images, are processedto generate a first combined image of the first scene area 430. In thoseembodiments in which the optional third image was captured optionalsub-step 428 is performed wherein the third image in addition to thefirst and second image is also processed to generate the first combinedimage of the scene area 430.

In some embodiments step 424 is accomplished using sub-step 426 whereinsaid processing of said first and second images and optionally saidthird image to generate a first combined image of the first scene areaincludes combining weighted pixel values of said first image, secondimage, and optional third image.

The weighting of the pixel values may, and in some embodiments is afunction of exposure times. Thus, at least in some embodiments, a pixelvalue of the combined image is generated by weighting and summing apixel value from each of the first, second and third images, where thepixel value from the first image is weighted according to the firstexposure time used to capture the first image, the pixel value from thesecond image is weighted according the second exposure time used tocapture the second image and the pixel value from the third image isweighted according to the third exposure time used to capture the thirdimage.

Operation proceeds from step 424 to step 432. In step 432, the generatedfirst combined image of the first scene area is stored indata/information 120 of memory 108 and/or displayed on the display 102,e.g., touch sensitive display of the camera device 100.

Operation proceeds from step 432 to step 404 where processing continuesand the method is repeated.

In some embodiments of the present invention step 424 is performed on anexternal device such as a computer that is coupled to the camera device100 via the transceiver interface 114. In such embodiments the first,second and optional third images are transmitted to the external devicevia the transceiver interface 114 where the step 424 is performed. Step432 is then typically performed by the external device with the combinedimage 430 being stored in memory associated with the external deviceand/or displayed on a display associated with the external device.

Method 400 may be, and in some embodiments is, implemented on a varietyof devices including for example, a camera or a mobile device such as amobile cellular telephone or a tablet.

In some embodiments, at least some of the optical chain modules includesingle color filters. For example, in one embodiment, the first opticalchain module includes a red filter, the second optical chain moduleincludes a green filter, the third optical chain module includes a bluefilter. In some such embodiments, at least two optical chain modules areprovided for each color for which a single color filter is used. Forexample in one embodiment, the plurality of optical chains modulesinclude two optical chain modules with a red filter, two optical chainmodules with a green filter and two optical chain modules with a bluefilter. In some embodiments, different optical chain modules havingsingle color filters corresponding to the same color have differentexposure times. In some embodiments, the combined image is generatedusing captured images of the first scene area from: (i) an optical chainmodule including a first color filter and a using first exposure time,(ii) an optical chain including a second color filter and using a firstexposure time, (iii) an optical chain including a third color filter andusing a first exposure time, (iv) an optical chain module including afirst color filter and a using second exposure time, (ii) an opticalchain including a second color filter and using a second exposure time,(iii) an optical chain including a third color filter and using a secondexposure time. In some such embodiments, the first color is red; thesecond color is green; and the third color is blue; the first exposuretime is a short exposure time and the second exposure time is a longexposure time.

In some embodiments, at least some optical chain modules do not includeany color filters.

Method 500 of FIG. 11 illustrates an embodiment of a method of producingat least one image of a first scene area in accordance with the presentinvention. The method 500 achieves enhanced sensor dynamic range bycombining images captured through the operation of two or more opticalchain modules using different exposure times. The processing steps ofthe method 500 of FIG. 11 will now be explained in view of the cameradevice 100 of FIG. 1A. For ease of explanation of the method 500, itwill be assumed that the plurality of optical chain module 130 of cameradevice 100 of FIG. 1A includes two optical chain modules and in someembodiments an optional third optical chain module which will bereferred to as a first, second and third optical chain modulerespectively. Method 500 is similar to method 400 but implements thecapture of the fourth image and display of the fourth image after thefirst, second and third images have been captured. In this way the userof the device is able to see on the display the first scene area thatwas captured in the first, second and optional third image and whichwill be processed to generate a combined image.

The method 500 of FIG. 11 starts at start step 502 with the start of thesteps of the method being implemented, e.g., on processor 110. Operationproceeds from start step 502 to step 504. In step 504, user input isreceived to control the capture of the image of the first scene area.The user input is received via input device 106 which may be, and insome embodiments is, a button or touch sensitive screen. From step 504processing proceeds to step 506 where the plurality of optical chainmodules 130 are operated in parallel to capture images of the firstscene area.

Step 506 includes sub-steps 510, 512, and optional sub-step 514. Insub-step 510, a first optical chain module is operated to capture afirst image 516 of the first scene area using a first exposure time. Insub-step 512, a second optical chain module is operated to capture asecond image 518 of the first scene area using a second exposure time,at least said first and said second exposure times being of differentduration but overlapping in time. In some embodiments, an optionalsub-step 514 is performed wherein a third optical chain module isoperated to capture a third image 520 of the first scene area using athird exposure time. In some embodiments, the third exposure time isdifferent than the first and second exposure times. Additional opticalchain modules may be, and in some embodiments are, used to captureadditional images of the first scene area with the additional opticalchain modules using the same or different exposure times as the first,second or third exposure times so as to obtain additional image data forthe first scene area and thereby enhancing the effective sensor dynamicrange of the camera device. Sub-steps 510, 512, and optional sub-step514 are performed in parallel so that multiple images of the first sceneare captured in parallel with different exposure times. The first,second and optional third captured images may be, and in someembodiments are, stored in data/information 120 of memory section 108 tobe available for later use such as for example in later steps of themethod for generating a combined image of the first scene area, or fordisplay or outputting of the images. Operation proceeds from step 506 tosteps 522 and 528.

In step 522, one of said first, second and optional third optical chainmodules is operated to capture a fourth image 524 of the first scenearea after capturing one of said first, second and third images. Whilein this particular embodiment the fourth image is captured after thefirst, second and third images, in some embodiments one of the first,second and third images is used as the fourth image. In some embodimentsa fourth exposure time different from said first, second and thirdexposure times is used to capture the fourth image 524. The fourth imagemay be, and in some embodiments is stored in data/information 120 ofmemory 108 for potential later use, output or display. Processingproceeds from step 522 to step 526. In step 526, the fourth image of thefirst scene area is displayed on display 102 of the camera device, e.g.,a touch sensitive screen so that a user of the camera device can see animage of the first scene area that was captured by the first, second andoptional third images. Processing proceeds from step 526 to step 504where processing associated with the method continues as the method isrepeated.

Returning to step 528, in step 528 the first and second images areprocessed to generate a first combined image of the first scene area534. In those embodiments in which the optional third image was capturedoptional sub-step 532 is performed wherein the third image in additionto the first and second images is also processed to generate the firstcombined image of the scene area 534.

In some embodiments step 528 is accomplished using sub-step 530 whereinsaid processing of said first and second images and optionally saidthird image to generate a first combined image of the first scene areaincludes combining weighted pixel values of said first image, secondimage, and optional third image. The weighting of the pixel values may,and in some embodiments is a function of exposure times. Thus, at leastin some embodiments, a pixel value of the combined image is generated byweighting and summing a pixel value from each of the first, second andthird images, where the pixel value from the first image is weightedaccording to the first exposure time used to capture the first image,the pixel value from the second image is weighted according the secondexposure time used to capture the second image and the pixel value fromthe third image is weighted according to the third exposure time used tocapture the third image.

Operation proceeds from step 528 to step 536. In step 536, the generatedfirst combined image of the first scene area is stored indata/information 120 of memory 108 and/or displayed on the display 102,e.g., the touch sensitive display of the camera device 100.

Operation proceeds from step 536 to step 504 where processing continuesand the method is repeated.

In some embodiments of the present invention step 528 is performed on anexternal device such as a computer that is coupled to the camera device100 via the transceiver interface 114. In such embodiments the first,second and optional third images are transmitted to the external devicevia the transceiver interface 114 where the step 528 is performed. Step536 is then typically performed by the external device with the combinedimage 534 being stored in memory associated with the external deviceand/or displayed on a display associated with the external device.

Method 500 may be, and in some embodiments, is implemented on a varietyof devices including for example, a camera or a mobile device such as amobile cellular telephone or a tablet.

The use of an external computer to perform some or a part of theprocessing of the first, second and optional third images allows for theuse of computational more complex algorithms as the external computermay be, and in some embodiments does have, a computationally morepowerful processing capability than the camera device 100.

In some embodiments, at least some of the optical chain modules includesingle color filters. For example, in one embodiment, the first opticalchain module includes a red filter, the second optical chain moduleincludes a green filter, the third optical chain module includes a bluefilter. In some such embodiments, at least two optical chain modules areprovided for each color for which a single color filter is used. Forexample in one embodiment, the plurality of optical chains modulesinclude two optical chain modules with a red filter, two optical chainmodules with a green filter and two optical chain modules with a bluefilter. In some embodiments, different optical chain modules havingsingle color filters corresponding to the same color have differentexposure times. In some embodiments, the combined image is generatedusing captured images of the first scene area from: (i) an optical chainmodule including a first color filter and a using first exposure time,(ii) an optical chain including a second color filter and using a firstexposure time, (iii) an optical chain including a third color filter andusing a first exposure time, (iv) an optical chain module including afirst color filter and a using second exposure time, (ii) an opticalchain including a second color filter and using a second exposure time,(iii) an optical chain including a third color filter and using a secondexposure time. In some such embodiments, the first color is red; thesecond color is green; and the third color is blue; the first exposuretime is a short exposure time and the second exposure time is a longexposure time.

In some embodiments, at least some optical chain modules do not includeany color filters. For example, in one particular embodiment, opticalchain modules OCM 171 and OCM 173 do not include color filters. Howeverin other embodiments, OCM 171 and OCM 173 each include a color filter.

Method 600 of FIG. 12 illustrates an embodiment of a method of producingat least one color image of a first scene area in accordance with thepresent invention. The method 600 uses color filters in connection withcombining two or more images of a first scene area to obtain a colorimage of the first scene area. The processing steps of the method 600 ofFIG. 12 will now be explained in view of the camera device 100 of FIG.1A. For ease of explanation of the method 600, it will be assumed thatthe plurality of optical chain module 130 of camera device 100 of FIG.1A includes two optical chain modules and in some embodiments anoptional third and/or fourth optical chain module which will be referredto as a first, second, third and fourth optical chain modulerespectively.

The method 600 of FIG. 12 starts at start step 602 with the start of thesteps of the method being implemented, e.g., on processor 110. Operationproceeds from start step 602 to step 604. In optional step 604, a fourthoptical chain module of the camera device is operated to capture animage, e.g., a image referred to herein as a fourth image of a firstscene area using a multi-color filter. This fourth image is capturedprior to capturing the first, second or third images which will bediscussed in connection with step 610 below.

Processing then proceeds to optional step 606 where the fourth image isdisplayed on the display 102 of the camera device 100. By displaying thefourth image on the display of the camera device 100 a user can aim thecamera device and target the first scene area for which the user wantsto capture an image. In some embodiments, the fourth image is alsostored in data/information 120 of memory 108. Processing then proceedsfrom step 606 to step 608.

In step 608, user input is received to control the capture of an imageof the first scene area. The user input is received via input device 106which may be, and in some embodiments is, a button or touch sensitivescreen. For example, the user may touch a portion of the touch sensitivescreen on which the fourth image is shown to focus the camera on aportion of the scene for which an image is to be captured. From step 608processing proceeds to step 610 where the plurality of optical chainmodules 130 are operated in parallel to capture images of the firstscene area.

Step 610 includes sub-steps 612, 614, and optional sub-step 616. Insub-step 612, a first optical chain module is operated to capture afirst image 618 of the first scene area using a first color filter. Insub-step 614, a second optical chain module is operated to capture asecond image 620 of the first scene area using a second color filter,said first and said second color filters corresponding to a first colorand a second color respectively. Said first and said second colors beingdifferent colors. In some embodiments, said first and second colorfilters are single color filters which correspond to said first andsecond colors, respectively. In some embodiments, an optional sub-step616 is performed wherein a third optical chain module is operated tocapture a third image 622 of the first scene area using a third colorfilter. In some embodiments, the third color filter corresponds to acolor that is different from said first and second colors. In someembodiments the third color filter is a single color filter whichcorresponds to said third color. Additional optical chain modules maybe, and in some embodiments are, used to capture additional images ofthe first scene area with the additional optical chain modules using thesame or different color filters as the first, second or third colorfilters so as to obtain additional image data for the first scene area.Sub-steps 612, 614, and optional sub-step 616 are performed in parallelso that multiple images of the first scene area are captured in parallelwith different color filters. The first, second and optional thirdcaptured images may be, and in some embodiments are, stored indata/information 120 of memory section 108 to be available for later usesuch as for example in later steps of the method for generating acombined image of the first scene area, or for display or outputting ofimages. In some embodiments of the present invention, the first opticalchain module includes a first lens and a first image sensor and thesecond optical module includes a second lens and a second image sensorand the optional third optical chain module includes a third lens and athird image sensor. In some embodiments, said first and said secondimage sensors are of the same resolution. In some embodiments of thepresent invention, said optional third image sensor of said thirdoptical chain module has the same resolution as the first and secondimage sensors. In some embodiments of the present invention, the fourthoptical chain module includes a fourth lens and a fourth image sensor.In some embodiments of the present invention the fourth image sensor isof the same resolution as the first and second image sensor. In someembodiments of the present invention, the first, second and third lensesof the first, second and third optical chain modules are arranged in acircle, and the fourth lens of the fourth optical chain is arranged inthe center of the circle.

Operation of the method proceeds from step 610 to step 624. In step 624the captured images, that is the first and second images, are processedto generate a first combined image of the first scene area 630. In thoseembodiments in which the optional third image was captured optionalsub-step 628 is performed wherein the third image in addition to thefirst and second images is also processed to generate the first combinedimage of the scene area 630. In some embodiments the fourth image of thefirst scene area is also processed with the first, second and thirdimages to generate the first combined image of the first scene area.

Operation proceeds from step 624 to step 632. In step 632, the generatedfirst combined image of the first scene area is stored indata/information 120 of memory 108 and/or displayed on the display 102,e.g., a touch sensitive display of the camera device 100.

Operation proceeds from step 632 to step 604 where processing continuesand the method is repeated.

In some embodiments of the present invention step 624 is performed on anexternal device such as a computer that is coupled to the camera device100 via the transceiver interface 114. In such embodiments the first,second and optional third images are transmitted to the external devicevia the transceiver interface 114 where the step 624 is performed. Step632 is then typically performed by the external device with the combinedimage 630 being stored in memory associated with the external deviceand/or displayed on a display associated with the external device.

Method 600 may be, and in some embodiments, is implemented on a varietyof devices including for example, a camera or a mobile device such as amobile cellular telephone or a tablet.

In some embodiments of the present invention, each image is presented asit is captured on the display or in the case of a combined image whensaid image has been generated.

In some embodiments of the present invention, each of the capturedimages, e.g., the first, second, third, and fourth images may be, andis, displayed on the display 102 of the camera device 100 as it iscaptured along with one or more combined images that are formed byprocessing and/or combining the first, second, third and/or fourthimages. In some embodiments of the present invention, each of the imagesmay be, is shown, in a separate portion of the display with the size ofthe image being adjusted so that each image displayed is shown in itsentirety. In some embodiments of the present invention, a caption isautomatically placed under each image as it displayed on the screen. Insome embodiments of the present invention, the caption includes thenumber of the image or an indication that it is a combined image, e.g.,image 1, image 2, image 3, image 4, combined image from image 1, 2, 3,and 4. In some embodiments of the present invention, each image ispresented as it is captured on the display or in the case of a combinedimage when said image has been generated. The images may be arranged ina variety of ways on the display 102 after capture and theaforementioned embodiments are only meant to be exemplary in nature.

In some embodiments of the present invention, the image generated bycombining the images captured from two or more of the optical chainmodules is displayed for targeting purposes so that the user may provideinput to control the capture of the image of the scene area and/or theobject in the scene upon which the combined image should be focused.

The FIG. 13 assembly of modules 1300 may, and in some embodiments is,used to process data for example first, second, third and fourth imagesand associated data, and storing and displaying images. Assembly ofmodules 1300 may be included in an exemplary apparatus, e.g., a cameradevice, e.g., camera device 100 of FIG. 1A, camera device 200 of FIG. 2,camera device 60 of FIG. 4, camera device 1500 of FIG. 15, camera device1605 of FIG. 16, camera device 1705 of FIG. 17, camera device 1801 ofFIG. 18, and/or camera device 1905 of FIG. 19, in accordance with anexemplary embodiment.

In some embodiments, assembly of modules 1300 is included in memory inan exemplary camera device, e.g., memory 108 of camera device 100 ofFIG. 1A, memory 213 of camera device 200 of FIG. 2, memory 73 of cameradevice 60 of FIG. 4, memory of camera device 1500 of FIG. 15, memory ofcamera device 1605 of FIG. 16, memory in camera device 1705 of FIG. 17,memory in camera device 1801 of FIG. 18, and/or memory of camera device1901 of FIG. 19. For example assembly of modules 1300 may be included aspart of assembly of modules 118 of memory 108 of camera device 100 ofFIG. 1.

In some embodiments, assembly of modules 1300 is implemented inhardware. In some embodiments, assembly of modules 1300 is implementedas software. In some embodiments, assembly of modules 1300 isimplemented as a combination of hardware and software.

In some embodiments, all or part of assembly of modules 1300 may beincluded as part of a processor, e.g., as part of processor 110 ofcamera device 100 of FIG. 1A.

In the FIG. 13 example, the assembly of modules 1300 includes a imageprocessing module 1302, a display module 1304, and a storage module1306. The modules implemented one or more of the previously discussedimage processing steps and may include a variety of sub-modules, e.g.,an individual circuit, for performing an individual step of method ormethods being implemented. Image processing module 1302 is configuredto: (1) process said first, second and third images to generate a firstcombined image of a first scene area, (2) receive user input indicatinga portion of the first scene area to be focused in the first combinedimage; and/or (3) shift pixel positions on at least one of said firstsecond and third images prior to generating said first combined image toalign the portion of the first scene area to be focused, as part ofprocessing said first, second, and third images to generate a firstcombined image; and (4) weight and sum a combination of pixel values ofthe first and second images corresponding to the same portion of thefirst scene area as a function of the first and second exposure timesrespectively and summing the weighted pixel values. In some embodimentsimage processing module 1302 is further configured to process said thirdimage to generate said first combined image of said first scene areafrom the third image in addition to said first and second images.

Display module 1304 is configured to display said fourth image on saiddisplay and configured to display said combined image on said display.Storage module 306 is configured to store or or more or said firstimage, said second image, said third image, said fourth image and saidcombined image in memory.

FIG. 14 illustrates a computer system which can be used for postprocessing of images captured using a camera device. The computer system1400 includes a display 1402, Input/Output (I/O) interface 1412,receiver 1404, input device 1406, transceiver interface 1414, processor1410 and memory 1408. Memory 1408 includes a first portion 1424including data/information 1420 and an assembly of modules 1418, and asecond portion 1426 including storage 1422. The memory 1408 is coupledto the processor 1410, I/O interface 1412 and transceiver interface 1414via bus 1416 through which the elements of the computer system 1400 canexchange data and can communicate with other devices via the I/Ointerface 1412 and/or interface 1414 which can couple the system 1400 toa network and/or camera apparatus. It should be appreciated that viainterface 1414 image data can be loaded on to the computer system 1400and subject to processing, e.g., post capture processing. The images maybe stored in the storage portion 1422 of memory 1408 for processing.Data/information 1420 includes, e.g., intermediate processing data andinformation and criteria used for processing e.g., weightinginformation, exposure time information, etc. The assembly of modules1418 includes one or more modules or routines which, when executed bythe processor 1410, control the computer system to implement one or moreof the image processing operations described in the present application.The output of multiple optical receiver chains can be, and in someembodiments is, combined to generate one or more images. The resultingimages are stored in the storage portion of the memory 1408 prior tobeing output via the network interface 1414, though another interface,or displayed on the display 1402. Thus, via the display 1402 a user canview image data corresponding to one or more individual optical chainmodules as well as the result, e.g., image, generated by combining theimages captured by one or optical chain modules.

FIG. 15 illustrates a frontal view of an apparatus 1500 implemented inaccordance with one embodiment of the present invention whichincorporates multiple optical chain modules. Camera device 1500 includesfour optical chains OCM 1 1502, OCM 2 1504, OCM 3 1506 and OCM 4 1508.The outer lens of OCM 1, OCM 2, OCM 3 and OCM 4, OCM 1 L1 1510, OCM 2 L11512, OCM 3 L1 1514, OCM 4 L1 1516, respectively, being shown as solidline circles with a frontal view. OCM 1 1502 including a red filterelement 1518, OCM 2 1504 including a green filter element 1520, OCM 31506 including a blue filter element 1522. Optical chain module 4 1508passes all three colors and includes a sensor with a multi-color filterelement 1524, e.g., a Bayer filter. The optical chain modules (1502,1504, 1506, 1508) may be the same as or similar to those previouslydescribed in FIGS. 1-3.

FIG. 16 illustrates a frontal view of the outer lenses of an apparatus1605, e.g., a camera device, implemented in accordance with oneembodiment of the present invention which incorporates multiple opticalchain modules and which is designed to have little or no gaps betweenthe outer most lenses of the different optical chain modules. The outermost lenses may be the aperture stop lenses in the FIG. 16 embodiment.Apparatus 1605 of FIG. 16 includes 7 optical chain modules OCM1, OCM2,OCM3, OCM4, OCMS, OCM6 and OCM7 with the outer lens plane correspondingto lenses L1 as viewed from the front of the camera device being shownin FIG. 16.

The 7 optical chain modules are, e.g., optical chain modules (OCM 1 161,OCM 2 161′, OCM 3 161″, . . . , OCM 7 161′″, of FIG. 1D with the outerlens (OCM 1 L1 162, OCM 2 L1 162′, OCM 3 L1 162″, . . . , OCM 7 L1162′″) being outer lenses (OCM 1 L1 1607, OCM 2 L1 1609, OCM 3 L1 1611,. . . , OCM 7 L1 1619) of FIG. 16, respectively.

The outer lenses L1 of optical chain modules 1, 2, 3, 4, 5, and 6, OCM 1L1 1607, OCM 2 L1 1609, OCM 3 L1 1611, OCM 4 L1 1613, OCM 5 L1 1615, OCM6 L1 1617, are positioned so as to surround the outer lens L1 of theoptical chain module 7, OCM 7 L1 1619. The outer lens L1 of the opticalchain module 7 1619 being formed in the shape of a hexagon, i.e., a sixsided polygon. The outer lenses L1 of optical chain modules 1, 2, 3, 4,5 and 6 (1607, 1609, 1611, 1613, 1615, 1617) being of same shape andsize and when combined with lens L1 of optical module 7 (1619) forming acircle. The optical center of each lens L1 of optical chain modules (OCM1 L1 1607, OCM 2 L1 1609, OCM 3 L1 1611, OCM 4 L1 1613, OCM 5 L1 1615,OCM 6 L1 1617) shown as a dark solid dot (1612, 1623, 1625, 1627, 1629,1631) on the dashed circle 1651. The optical center of lens L1 1619 ofoptical chain module 7 shown as a dot 1633 in the center of the hexagonand also in center of the dashed line 1651. A block separator or otherlight block may be used between the lenses to stop light leakage betweenthe different lenses. The dots (1621, 1623, 1625, 1627, 1629, 1631,1633) in FIG. 16 represent the optical center of the individual lenses(1607, 1609, 1611, 1613, 1615, 1617, 1619), respectively. In someembodiments each outermost lens is a round convex lens with itsparameter cut to the shape shown in FIG. 16 so that the lenses fightclosely together. The little or no gap between the front lenses, e.g.,the total area of the gap between the lenses occupies less than 5% ofthe total area of the front area of the lens assembly, e.g., circleshown in FIG. 16, occupied by the lenses when assembled together. Thelack of or small size of the gaps facilitates generating combined imageswith a desirable bokehs or blurs in the combined image with regard toimage portions which are out of focus, e.g., in some cases without theneed for extensive and potentially complex processing to generate thecombined image.

In FIG. 16, circle 1603 represents a circular aperture for the cameradevice 1605. In other embodiments, the aperture for the camera device1605 is a polygon shaped aperture. The plurality of lenses (1607, 1609,1611, 1613, 1615, 1615, 1617, 1619) are configured to partition theaperture 1603 into a plurality of light capture areas (1641, 1643, 1645,1647, 1649, 1651, 1653), occupying substantially the entire area of thefirst aperture.

In some embodiments, the seven optical chains included in camera device1605 are the N optical chains (161, 161′, 161″ . . . , 161′″), whereN=7, where the outer lenses configuration of FIG. 16 is used. Forexample, OCM 1 L1 162 of FIG. 1D is OCM L1 1607 of FIG. 16, OCM 2 L1162′ of FIG. 1D is OCM 2 L1 1609 of FIG. 16, OCM 3 L1 162″ of FIG. 1D isOCM 3 L1 1611 of FIG. 16, . . . , and OCM N L1 162′″ of FIG. 1D is OCM 7L1 1619 of FIG. 16.

In various embodiments, the sensor included in each optical chain incamera device 1605 is a semiconductor sensor. In various embodiments,first aperture of camera device 1605 is one of a circular or polygonshaped aperture. The first aperture of camera device 1605 corresponds tocircle 1603. In some other embodiments, the first aperture correspondsto a polygon, e.g., a polygon approximately the same size as circle1603. In some embodiments, the polygon fits inside circle 1603. In someembodiments, the polygon is a regular polygon.

The lenses (1607, 1609, 1611, 1613, 1615, 1617) in said plurality oflenses (1607, 1609, 1611, 1613, 1615, 1617, 1619) which are arrangedalong the perimeter of said first aperture 1603 have optical centers(1621, 1623, 1625, 1627, 1629, 1631) which are arranged along a circle1651. The lenses (1607, 1609, 1611, 1613, 1615, 1617) in said pluralityof lenses (1607, 1609, 1611, 1613, 1615, 1617, 1619) which are arrangedalong the perimeter of said first aperture 1603 have optical centers(1621, 1623, 1625, 1627, 1629, 1631) which form the vertices (corners)of a regular polygon 1655.

The plurality of lenses (1607, 1609, 1611, 1613, 1615, 1617, 1619)includes at least one inner lens 1619 in addition to said lenses (1607,1609, 1611, 1613, 1615, 1617) arranged along the perimeter of said firstaperture 1603. The plurality of lenses (1607, 1609, 1611, 1613, 1615,1617, 1619) includes a total of six lenses (1607, 1609, 1611, 1613,1615, 1617) along the perimeter of said first aperture 1603 and a singlelens (1619) in the center of said six lenses (1607, 1609, 1611, 1613,1615, 1617) arranged along the perimeter of said first aperture 1603.

The non-circular aperture of each of said plurality of lenses (1607,1609, 1611, 1613, 1615, 1617, 1619) is an aperture stop in acorresponding optical chain.

Each lens in said plurality of lenses (1607, 1609, 1611, 1613, 1615,1617, 1619) is part of a corresponding optical chain, each individualoptical chain includes a separate sensor for capturing an imagecorresponding to said individual optical chain.

Apparatus 1605, e.g., a camera device, further includes a module, e.g.,module 1302 of FIG. 13, for combining images captured by separateoptical chains into a single combined image. In various embodiments, thecombining images, e.g., performed by module 1302, includes a shift andadd based on the position of lenses in said plurality of lenses (1607,1609, 1611, 1613, 1615, 1617, 1619).

Camera device 1605 further includes additional elements shown in FIG. 1Aincluding a processor, a memory and a display.

FIG. 17 illustrates a frontal view of the outer lenses of an apparatus1705 implemented in accordance with one embodiment of the presentinvention which incorporates multiple optical chain modules and outerlenses, e.g., the aperture stop lens for each of the correspondingoptical chains, arranged to have non-uniform spacing between the opticalcenters of the lenses. Thus the FIG. 17 embodiment is similar to theFIG. 16 embodiment but with non-uniform spacing of the optical centersof lenses along the outer parameter of the lens assembly. Thenon-uniform spacing facilitates depth of field determinationsparticularly when performing block processing and the entire field ofview may not be under consideration when processing a block orsub-portion of the captured field of view. The optical chain modulesshown in FIGS. 16 and 17 are the same or similar to those previouslydescribed with reference to FIG. 3 but differ in terms of lens shape,size and/or configuration. The dots (1721, 1723, 1725, 1727, 1729, 1731,1733) in FIG. 17 represent the optical center of the individual lenses(1707, 1709, 1711, 1713, 1715, 1717, 1719), respectively.

FIG. 18 illustrates another exemplary camera device 1801 including aplurality of first through fifth optical chain modules (1890, 1891,1892, 1893, 1894) each of which includes an outer lens (1813, 1815,1817, 1819, 1821), respectively, represented as a circle on the outerlens platter 1803. Each outer lens (1813, 1815, 1817, 1819, 1821) has anoptical axis (1805, 1806, 1807, 1808, 1809), respectively. The opticalaxis (1805, 1806, 1807, 1808, 1809) is represented by an X, indicatingthat the axis goes down into the lens (1813, 1815, 1817, 1819, 1821).The optical axis (1805, 1806, 1807, 1808, 1809), are parallel to eachother. In both FIGS. 18 and 19 arrows made of dashed lines represent thepath of light for the corresponding optical chain module after lightwhich entered the outer lens along the optical axis of the outer lens isredirected by the mirror or other light redirection device. Thus, thearrows represents the direction and general light path towards thesensor of the optical chain to which the arrow corresponds. In variousembodiments, the image deflection element, e.g., a mirror, of theoptical chain changes the direction of the optical rays passing alongthe optical axis of the outer lens by substantially 90 degrees to directthe optical rays passing along the optical axis onto the sensor. Forexample, with regard to optical chain 1890, the image deflection element1823, e.g., a mirror, of the optical chain 1890 changes the direction ofthe optical rays passing along the optical axis 1805 of the outer lens1813 by substantially 90 degrees to direct the optical rays passingalong the optical axis onto the sensor 1853.

In the FIG. 18 embodiment each of the optical chain modules (1890, 1891,1892, 1893, 1894) includes, in addition to an outer lens (1813, 1815,1817, 1819, 1821,) a mirror or other device, e.g., prism, (1823, 1825,1827, 1829, 1831), respectively, for changing the angle of lightreceived via the corresponding outer lens (1813, 1815, 1817, 1819,1821), respectively. Additionally, as in some of the previouslydescribed embodiments such as the FIGS. 1A, 1B, 1C, 1D, and 3embodiments, each optical chain module (1890, 1891, 1892, 1893, 1894),includes a filter (1833, 1835, 1837, 1839, 1841), respectively, and aninner lens (1843, 1845, 1847, 1849, 1851), respectively. In additioneach optical chain module (1890, 1891, 1892, 1893, 1894) includes asensor (1853, 1855,1857, 1859, 1861), respectively. For example, thefirst optical chain module (OCM 1 1890) include outer lens L1 1813,mirror 1823, filter 1833, inner lens L2 1843 and sensor 1853.

Filters 1833, 1835, 1837, 1839, and 1841 are mounted on a movablecylinder 1875 represented as a circle shown using small dashed lines.The cylinder 1875 may be rotated and/or moved forward or backwardallowing lenses and/or filters on the cylinder to be easily replacedwith other lenses, filter, or holes mounted on the cylinder 1875. Whilein the FIG. 18 example, an exit hole is provided to allow light to exitcylinder 1875 after passing through one of the filters 1833, 1835, 1837,1839, or 1841 it should be appreciated that rather than an exit holeanother lens or filter may be mounted on the cylinder 1875 allowing twoopportunities for the light to be filtered and/or passed through a lensas is passes through the cylinder 1875. Thus, in at least someembodiments a second filter or lens which is not shown in FIG. 18 forsimplicity is included at the exit point for the light as it passesthrough cylinder 1804. Inner lenses are mounted on cylinder 1885 whichis actually closer to the outside sidewalls of the camera device 1801than the filters mounted on cylinder 1875. Given the large diameter ofmovable cylinder 1885 and the relatively small diameter of the lightbeam as it nears the sensor, it should be appreciated that a largenumber of alternative filters, lenses and/or holes can be mounded oncylinder 1885. As with cylinder 1875 the light can be filtered and/orprocessed by a lens as it enters and leaves cylinder 1885 prior toreaching the sensor of the corresponding optical chain.

In some embodiments lenses mounted on a moveable platter positionedbetween the outer lens platter 1803 and mirrors which may, and in someembodiments are, also mounted on a platter are used to supportautofocus. In such an embodiment the lens platter between the outer lensplatter and mirror platter is moved in or out to perform focusoperations for each of the optical chain modules in parallel. In anotherembodiment, different sets of lens are mounted on the drum 1885 or 1875with different lens sets being mounted with a different offset distancefrom the surface of the drum. By switching between the different sets oflenses by rotating the drum on which the different lens sets aremounted, focusing between different predetermined focus set points can,and in some embodiments is achieved, by simply rotating the drum onwhich the lens sets, corresponding to the different focal distance setpoints, are mounted.

Notably, the FIG. 18 embodiment, by changing the direction of lightthrough the use of mirrors, prisms and/or other devices allows for thelength of the individual optical chains to be longer than the cameradevice is thick. That is, the side to side length of the camera device1801 can be used in combination with a portion of the front to backlength to create optical chains having a length longer than the depth ofthe camera device 1801. The longer optical chain length allows for morelenses and/or filters to be used as compared to what may be possiblewith shorter optical chain lengths. Furthermore, the change in thedirection of light allows for the use of cylinders for mounting lenses,filters and/or holes which can be easily interchanged by a simplerotation or axial, e.g., front to back movement, of the cylinder onwhich the lenses, filters and/or holes corresponding to multiple opticalchains are mounted.

In the FIG. 18 embodiment sensors may be fixed and/or mounted on amovable cylinder 1899. Thus, not only can the lenses, filters and/orholes be easily switched, changes between sensors or sets of sensor canbe easily made by rotating the cylinder on which the sensors aremounted. While a single mirror is shown in FIG. 18 in each optical chainmodule, additional mirrors may be used to further extend the length ofthe optical path by reflecting in yet another direction within thehousing of the camera device 1801.

It should be appreciated that the FIG. 18 embodiment allows for acombination of lens, filter, and/or hole mounting platters arrangedparallel with the platter extending left to right within the cameradevice and cylinders arranged so that the top and bottom of the cylinderextend in the front to back direction with respect to the camera body,e.g., with the front of the camera being shown in FIG. 18. Cylinders maybe mounted inside of one another providing a large number ofopportunities to mount lens, filters and/or holes along the opticalpaths of each optical chain module and allowing for a large number ofpossible filter/lens/sensor combinations to be supported, e.g., byallowing for different combinations of cylinder positions for differentmodes of operation.

While changing sensors mounted on a cylinder can be achieved by rotatinga cylinder, in the earlier embodiments in which sensors may be mountedon platters, sensors may be changed by rotating or otherwise moving aplatter on which the sensors are mounted.

Note that in the FIG. 18 embodiment the outer lenses (1813, 1815, 1817,1819, 1821, of the optical chain modules (1890, 1891, 1892, 1893, 1894),respectively, are mounted near the center of the front of the cameradevice 1801 as shown, e.g., forming a generally circular pattern ofouter lenses 1813, 1815, 1817, 1819, 1821.

In camera device 1801 the optical axes (1805, 1806, 1807, 1808, 1809) oflenses (1813, 1815, 1817, 1819, 1821) said optical chain modules (1890,1891, 1892, 1893, 1894) are parallel to each other but at least twomirrors (1823, 1825) corresponding to different optical chains (1890,1891) are not parallel. The light rays of at least two different opticalchains (1890, 1891) cross prior to reaching the sensor (1853, 1855) towhich the rays of said at least two different optical chain modules(1890, 1891) correspond.

In various embodiments, each optical chain module (1890, 1891, 1892,1893, 1894) includes an image deflection element which includes at leastone mirror positioned at 45 degree to said optical axis (1890, 1891,1892, 1893, 1894) of said lens of the optical chain module. For example,with regard to optical chain module 1 1890, in one embodiments, theimage deflection element 1823 is a mirror positioned at 45 degree to theoptical axis 1805 of lens 1813.

In some embodiments, an image deflection element, e.g., image deflectionelement 1823 includes a prism. In some embodiments, an image deflectionelement includes multiple mirrors. In some embodiments, an imagedeflection element includes a combination including at least one mirrorand at least one prism.

FIG. 19 is similar to the FIG. 18 embodiment in that it illustratesanother camera device 1901 including a plurality of optical chainmodules which include mirrors or another device for changing the angleof light entering the optical chain module and thereby allowing at leasta portion of the optical chain module to extend in a direction, e.g., aperpendicular direction, which is not a straight front to back directionwith respect to the camera device. FIG. 19 illustrates another exemplarycamera device 1901 including a plurality of first through fifth opticalchain modules (1990, 1991, 1992, 1993, 1994) each of which includes anouter lens (1913, 1915, 1917, 1919, 1921), respectively, represented asa circle on the outer lens platter 1903. FIG. 19 differs from the FIG.18 embodiment in that the outer lenses (1913, 1915, 1917, 1919, 1921) ofthe first through fifth optical chain modules (1990, 1991, 1992, 1993,1994) are positioned near the perimeter of the face of the camera device1901. This allows for the length of the optical chain module to belonger than the length of the optical chains shown in FIG. 18. FIG. 19shows outer and inner cylinders, also some times referred to as drums,1975, 1985, upon which filters, lenses and holes can and in variousembodiments are mounted as discussed with regard to the FIG. 18embodiment. Thus cylinders 1975 and 1985 server the same or similarpurpose served by cylinders 1875, 1885, respectively. It should beappreciated that in some embodiments the FIG. 19 embodiment includesfilters and lenses mounted on the inner and outer cylinders in the sameor similar manner as filters and lenses are mounted on the cylinders1875, 1885 shown in FIG. 18.

Elements of the FIG. 19 embodiment which are the same or similar to theelements of the FIG. 18 embodiment are identified beginning with “19”instead of “18” and for the sake of brevity will not be described againin detail. For example element 1961 is used to refer to the sensor forthe optical chain module 1994 which includes outer lens 1921,mirror/light redirection device 1931, filter 1941 and inner lens 1951.The cylinder 1975 is used to mount the filters while cylinder 1985 isused to mount the inner lenses.

Each outer lens (1913, 1915, 1917, 1919, 1921) has an optical axis(1905, 1906, 1907, 1908, 1909), respectively. The optical axis (1905,1906, 1907, 1908, 1909) is represented by an X, indicating that the axisgoes down into the lens (1913, 1915, 1917, 1919, 1921). The optical axis(1905, 1906, 1907, 1908, 1909), are parallel to each other.

The camera devices 1801 and 1901 may, and in some embodiments do,include a processor, display and/or other components of the cameradevice shown in FIG. 1A but such elements are not explicitly shown inthe FIGS. 18 and 19 embodiments to avoid complicating the figures andbeing repetitive.

Various functions of the present invention may be and are implemented asmodules in some embodiments. The assembly of modules 1300 shown in FIG.13 illustrates an exemplary assembly of modules, e.g., software orhardware modules, that may be and are used for performing variousfunctions of a image processing system or apparatus used to processimages in accordance with embodiments of the present invention. When themodules identified in FIG. 13 are implemented as software modules theymay be, and in some embodiments of the present invention are, stored inmemory 108 of FIG. 1A in the section of memory identified as assembly ofmodules 118. These modules may be implemented instead as hardwaremodules, e.g., circuits.

The ideas and concepts described with regard to various embodiments suchas those shown in FIG. 19 can be extended so that the input sensors canbe located in a plane, e.g., at the back of the camera device and/or atthe front of the camera device. In some such embodiments the sensors ofmultiple optical chains are mounted on a flat printed circuit board orbackplane device. The printed circuit board, e.g. backplane, can bemounted or coupled to horizontal or vertical actuators which can bemoved in response to detected camera motion, e.g., as part of a shakecompensation process which will be discussed further below. In some suchembodiments, pairs of light diverting devices, e.g., mirrors, are usedto direct the light so that at least a portion of each optical chainextends perpendicular or generally perpendicular to the input and/orsensor plane. Such embodiments allow for relatively long optical pathswhich take advantage of the width of the camera by using mirrors orother light diverting devices to alter the path of light passing throughan optical chain so that at least a portion of the light path extends ina direction perpendicular or generally perpendicular to the front of thecamera device. The use of mirrors or other light diverting devicesallows the sensors to be located on a plane at the rear or front of thecamera device as will now be discussed in detail.

While the invention has been explained using convex lenses in many ofthe diagrams, it should be appreciated that any of a wide variety ofdifferent types of lenses may be used in the optical chain modulesincluding, e.g., convex, concave, and meniscus lenses. In addition,while lenses and filters have been described as separate elements,lenses and filters may be combined and used. For example, a color lensmay, and in some embodiments is, used to both filter light and alter thelights path. Furthermore, while many of the embodiments have beendescribed with a color filter preceding the image sensor of an opticalchain or as using an image sensor with an integrated color filter, e.g.,a Bayer pattern filter, it should be appreciated that use of colorfilters and/or sensors with color filters is not required and in someembodiments one or more optical chain modules are used which do notinclude a color filter and also do not use a sensor with a color filter.Thus, in some embodiments one or more optical chain modules which sensea wide spectrum of color light are used. Such optical chain modules areparticularly well suited for generating black and white images.

In various embodiments image processing is used to simulate a widevariety of user selectable lens bokehs or blurs in the combined imagewith regard to image portions which are out of focus. Thus, whilemultiple lenses are used to capture the light used to generate acombined image, the image quality is not limited to that of anindividual one of the lenses and a variety of bokehs can be achieveddepending on the particular bokeh desired for the combined image beinggenerated. In some embodiments, multiple combined images with differentsimulated bokehs are generated using post image capture processing withthe user being provided the opportunity to save one or more of thegenerated combined images for subsequent viewing and/or printing. Thus,in at least some embodiments a physical result, e.g., a printed versionof one or more combined images is produced. In many if not all casesimages representing real world objects and/or scenes which were capturedby one or more of the optical chain modules of the camera device used totake the picture are preserved in digital form on a computer readablemedium, e.g., RAM or other memory device and/or stored in the form of aprinted image on paper or on another printable medium.

While explained in the context of still image capture, it should beappreciated that the camera device and optical chain modules of thepresent invention can be used to capture video as well. In someembodiments a video sequence is captured and the user can select anobject in the video sequence, e.g., shown in a frame of a sequence, as afocus area, and then the camera device capture one or more images usingthe optical chain modules. The images may, and in some embodiments are,combined to generate one or more images, e.g., frames. A sequence ofcombined images, e.g., frames may and in some embodiments is generated,e.g., with some or all individual frames corresponding to multipleimages captured at the same time but with different frames correspondingto images captured at different times.

While different optical chain modules are controlled to use differentexposure times in some embodiments to capture different amounts of lightwith the captured images being subsequently combined to produce an imagewith a greater dynamic range than might be achieved using a singleexposure time, the same or similar effects can and in some embodimentsis achieved through the use of different filters on different opticalchains which have the same exposure time. For example, by using the sameexposure time but different filters, the sensors of different opticalchain modules will sense different amounts of light due to the differentfilters which allowing different amounts of light to pass. In one suchembodiment the exposure time of the optical chains is kept the same byat least some filters corresponding to different optical chain modulescorresponding to the same color allow different amounts of light topass. In non-color embodiments neutral filters of different darknesslevels are used in front of sensors which are not color filtered. Insome embodiments the switching to a mode in which filters of differentdarkness levels is achieved by a simple rotation or movement of a filterplatter which moves the desired filters into place in one or moreoptical chain modules. The camera devices of the present inventionsupports multiple modes of operation with switching between panoramicmode in which different areas are captured, e.g., using multiple lensesper area, and a normal mode in which multiple lens pointed samedirection are used to capture the same scene. Different exposure modesand filter modes may also be supported and switched between, e.g., basedon user input.

FIG. 20 is a drawing of an assembly of modules 2000 in accordance withan exemplary embodiment. Assembly of modules 2000 may be included in anexemplary apparatus, e.g., a camera device, e.g., camera device 100 ofFIG. 1A, camera device 200 of FIG. 2, camera device 60 of FIG. 4, cameradevice 1500 of FIG. 15, camera device 1605 of FIG. 16, camera device1705 of FIG. 17, camera device 1801 of FIG. 18 and/or camera device 1901of FIG. 19, in accordance with an exemplary embodiment.

In some embodiments, assembly of modules 2000 is included in memory inan exemplary camera device, e.g., memory 108 of camera device 100 ofFIG. 1A, memory 213 of camera device 200 of FIG. 2, or memory 73 ofcamera device 60 of FIG. 4, memory in camera device 1500 of FIG. 15,memory in camera device 1605 of FIG. 16, memory in camera device 1705 ofFIG. 17, memory in camera device 1801 of FIG. 18 and/or memory cameradevice 1901 of FIG. 19. For example assembly of modules 2000 may beincluded as part of assembly of modules 118 of memory 108 of cameradevice 100 of FIG. 1.

In some embodiments, assembly of modules 2000 is included in anexemplary device, e.g., an exemplary camera device, which implements amethod in accordance with flowchart 900 of FIG. 9.

Assembly of modules 2000 includes a module 2004 configured to receiveuser input to control capture of at least one image of a first scene,and a module 2008 configured to operate a plurality of three or moreoptical chain modules in parallel to capture images of a first scenearea, said images including at least of a first image of said firstscene area, a second image of the first scene area, and a third image ofthe first scene area. Module 2004 includes a module 2006 configured toreceive user input indicating a portion of the first scene area to befocused. Assembly of modules 2008 includes a module 2010 configured tooperate a first optical chain module to capture a first image of thefirst scene area, a module 2018 configured to operate a second opticalchain module to capture a second image of the first scene area, and amodule 2014 configured to operate a third optical chain module tocapture a third image of the first scene area.

Assembly of modules 2000 further includes a module 2016 configured tostore the captured first image of the first scene area, a module 2018configured to store the captured second image of the first scene area,and a module 2020 configured to store the captured third image of thefirst scene area. Assembly of modules 2000 further includes a module2022 configured to process the first, second, and third images togenerate a first combined image of the first scene area to be focused.Module 2022 includes a module 2024 configured to shift pixel portions ofat least one of the first, second, and third images to align the portionof the first scene area to be focused. Assembly of modules 2000 furtherincludes a module 2028 configured to store in memory the combined imageand a module 2029 configured to display the combined image on a display.

FIG. 21 is a drawing of an assembly of modules 2100 in accordance withan exemplary embodiment. Assembly of modules 2100 may be included in anexemplary apparatus, e.g., a camera device, e.g., camera device 100 ofFIG. 1A, camera device 200 of FIG. 2, camera device 60 of FIG. 4, cameradevice 1500 of FIG. 15, camera device 1605 of FIG. 16, camera device1705 of FIG. 17, camera device 1801 of FIG. 18, camera device 1901 ofFIG. 19, in accordance with an exemplary embodiment.

In some embodiments, assembly of modules 2100 is included in memory inan exemplary camera device, e.g., memory 108 of camera device 100 ofFIG. 1A, memory 213 of camera device 200 of FIG. 2, or memory 73 ofcamera device 60 of FIG. 4, memory in camera device 1500 of FIG. 15,memory in camera device 1605 of FIG. 16, memory in camera device 1705 ofFIG. 17, memory in camera device 1801 of FIG. 18, memory in cameradevice 1901 of FIG. 19. For example assembly of modules 2100 may beincluded as part of assembly of modules 118 of memory 108 of cameradevice 100 of FIG. 1.

In some embodiments, assembly of modules 2100 is included in anexemplary device, e.g., an exemplary camera device, which implements amethod in accordance with flowchart 1000 of FIG. 10.

Assembly of module 2100 includes a module 2104 configured to operate oneof a plurality of optical chain modules to capture an image, e.g., afourth image, of a first scene area, a module 2105 configured to displaythe fourth image on a display, a module 2107 configured to store thefourth image of the first scene area, a module 2108 configured toreceive user input to control capture of an image of the first scenearea. Assembly of modules 2100 further includes a module 2110 configuredto operate the plurality of optical chain modules in parallel to captureimages of the first scene area. Module 2110 includes a module 2112configured to operate a first optical chain module to capture a firstimage of the first scene area using a first exposure time, a module 2114configured to operate a second optical chain module to capture a secondimage of the first scene area using a second exposure time, and a module2116 configured to operate a third optical chain module to capture athird image of the first scene area using a third exposure time.Assembly of modules 2100 further includes a module 2118 configured tostore the captured first image of the first scene area, a module 2120configured to store the captured second image of the first scene area,and a module 2122 configured to store the captured third image of thefirst scene area.

Assembly of modules 2100 further includes a module 2124 configured toprocess the images to generates a first combined image of the firstscene area. Module 2124 includes a module 2126 configured to weight andsum a combination of pixel values of the images as a function ofexposure time including, e.g., weighting pixel values of the first andsecond images corresponding to the same portion of the first scene areaas a function of the first and second exposure times, respectively, andsumming the weighted pixel values, and a module 2128 configured tooptionally process the third image in addition to the first and secondimages to generate the first combined image. Assembly of modules 2100further includes a module 2132 configured to store in memory thecombined image and a module 2133 configured to display the combinedimage on a display.

FIG. 22 is a drawing of an assembly of modules 2200 in accordance withan exemplary embodiment. Assembly of modules 2200 may be included in anexemplary apparatus, e.g., a camera device, e.g., camera device 100 ofFIG. 1A, camera device 200 of FIG. 2, camera device 60 of FIG. 4, cameradevice 1500 of FIG. 15, camera device 1605 of FIG. 16, camera device1705 of FIG. 17, camera device 1801 of FIG. 18 and/or camera device 1901of FIG. 19, in accordance with an exemplary embodiment.

In some embodiments, assembly of modules 2200 is included in memory inan exemplary camera device, e.g., memory 108 of camera device 100 ofFIG. 1A, memory 213 of camera device 200 of FIG. 2, memory 73 of cameradevice 60 of FIG. 4, memory of camera device 1500 of FIG. 15, memory ofcamera device 1605 of FIG. 16, memory of camera device 1705 of FIG. 17,memory of camera device 1801 of FIG. 18, and/or memory of camera device1901 of FIG. 19. For example assembly of modules 2100 may be included aspart of assembly of modules 118 of memory 108 of camera device 100 ofFIG. 1.

In some embodiments, assembly of modules 2200 is included in anexemplary device, e.g., an exemplary camera device, which implements amethod in accordance with flowchart 1100 of FIG. 11.

Assembly of module 2200 includes a module 2204 configured to receiveuser input to control capture of an image of the first scene area, and amodule 2206 configured to operate the plurality of optical chain modulesin parallel to capture images of the first scene area. Module 2206includes a module 2210 configured to operate a first optical chainmodule to capture a first image of the first scene area using a firstexposure time, a module 2212 configured to operate a second opticalchain module to capture a second image of the first scene area using asecond exposure time, and a module 2214 configured to operate a thirdoptical chain module to capture a third image of the first scene areausing a third exposure time. Assembly of modules 2200 further includes amodule 2216 configured to store the captured first image of the firstscene area, a module 2218 configured to store the captured second imageof the first scene area, and a module 2220 configured to store thecaptured third image of the first scene area. Assembly of modules 2200further includes a module 2222 configured to operate one of the firstsecond or third optical chain modules to capture an image, e.g., afourth image of a first scene area, a module 2224 configured to storethe captured image, e.g., the captured fourth image, of the first scenearea, and a module 2226 configured to display the fourth image of thefirst scene area on a display.

Assembly of modules 2200 further includes a module 2228 configured toprocess the first and second images to generate a first combined imageof the first scene area. Module 2228 includes a module 2130 configuredto weight and sum a combination of pixel values of the images as afunction of exposure time including, e.g., weighting pixel values of thefirst and second images corresponding to the same portion of the firstscene area as a function of the first and second exposure times,respectively, and summing the weighted pixel values, and a module 2232configured to optionally process the third image in addition to thefirst and second images to generate the first combined image. Assemblyof modules 2200 further includes a module 2236 configured to store inmemory the combined image and a module 2237 configured to display thecombined image on a display.

FIG. 23 is a drawing of an assembly of modules 2300 in accordance withan exemplary embodiment. Assembly of modules 2300 may be included in anexemplary apparatus, e.g., a camera device, e.g., camera device 100 ofFIG. 1A, camera device 200 of FIG. 2, camera device 60 of FIG. 4, cameradevice 1500 of FIG. 15, camera device 1605 of FIG. 16, camera device1705 of FIG. 17, camera device 1801 of FIG. 18, and/or camera device1901 of FIG. 19, in accordance with an exemplary embodiment.

In some embodiments, assembly of modules 2300 is included in memory inan exemplary camera device, e.g., memory 108 of camera device 100 ofFIG. 1A, memory 213 of camera device 200 of FIG. 2, memory 73 of cameradevice 60 of FIG. 4, memory of camera device 1500 of FIG. 15, memory ofcamera device 1605 of FIG. 16, memory of camera device 1705 of FIG. 17,memory of camera device 1801 of FIG. 18, and/or memory of camera device1901 of FIG. 19. For example assembly of modules 2100 may be included aspart of assembly of modules 118 of memory 108 of camera device 100 ofFIG. 1.

In some embodiments, assembly of modules 2300 is included in anexemplary device, e.g., an exemplary camera device, which implements amethod in accordance with flowchart 1200 of FIG. 12.

Assembly of module 2300 includes a module 2304 configured to operate afourth optical chain module to capture an image, e.g., a fourth image,of a first scene area, using a multi-color filter, a module 2306configured to display the fourth image on a display, a module 2308configured to receive user input to control capture of an image of thefirst scene area, and a module 2310 configured to operate the pluralityof optical chain modules in parallel to capture images of the firstscene area. Module 2310 includes a module 2312 configured to operate afirst optical chain module to capture a first image of the first scenearea using a first color filter, a module 2314 configured to operate asecond optical china module to capture a second image of the first scenearea using a second color filter, and a module 2316 configured tocapture a third image of the first scene area using a third colorfilter. Assembly of modules 2300 further includes a module 2318configured to store the captured image of the first scene area, a module2320 configured to store the captured second image of the first scenearea, and a module 2322 configured to store the captured third image ofthe first scene area. Assembly of modules 2300 further includes a module2324 configured to process the first and second images to generate afirst combined image of the first scene area. Module 2324 includes amodule 2328 configured to process the third image in addition to thefirst and second images to generate the first combined image.

Assembly of modules 2300 further includes a module 2332 configured tostore in memory the combined image and a module 2333 configured todisplay the combined image on a display. In some embodiments, anexemplary camera device, e.g., camera device 100 of FIG. 1, includes oneor more of all of assembly of modules 1300, assembly of modules 2000,assembly of modules 2100, assembly of modules 2200, and assembly ofmodules 2300. An assembly of modules may be implemented in hardware,software, or a combination of hardware and software, e.g., dependingupon the particular embodiment.

FIG. 24, which comprise the combination of FIGS. 24A and 24B,illustrates an exemplary method 2400 for generating pixel values of acombined image from pixel values generated by a plurality of opticalchain modules operating in parallel. The method 2400 begins with theprocessor implementing the method, e.g., the processor 110 of the cameradevice 100, or processor 1410 of the post-processing system 1400,beginning the process of generating a combined image.

In step 2404 pixel values corresponding to the same scene, e.g., pixelvalues generated by optical chain modules operating in parallel,generated by multiple optical camera modules, e.g., optical chainmodules 161, 161′ and 161″, are received. The receipt may be the resultof the processor 110 or 1410 reading the values from memory or receivingthem directly from the OCMs which generated the values.

In step 2406 the pixel values are grouped according to type. Forexample, some OCMs may provide R (red) pixel values, some blue (B) pixelvalues others (G) green pixel values while still others may provideunfiltered pixel value indicative of luminance (L) resulting frommultiple colors of light reaching the sensor 168, 168′ or 161′″ of theoptical chain module which generated the pixel values to be processed.In addition to the type of pixel value being processed the processor isaware of the exposure time used by the optical chain module to generatethe pixel value. This information may be know to the processor if itcontrolled the exposure time, or from information stored with the pixelvalues and supplied to the processor along with the pixel values andinformation indicating the type of pixel values being supplied as wellas, in some cases, other useful information such as the configurationand location (lens spacing) of OCMs which were the source of pixelvalues.

With the pixel values being separated according to type, processing ofthe different types of pixel values may proceed with pixel values of agiven type from different OCMs being processed and combined to generatepixel values of the combined image.

Steps 2408 through 2422 are performed for each type of pixel value to beprocessed, e.g., with R, G, B values being processed separately.Similarly in the case of Luminance values (unfiltered) values suchvalues are treated as a separate set of pixel values for processingpurposes and may be used for generating a grayscale image or incombination with color information at rendering time when an image is tobe displayed.

Combining of pixel values of an individual type beings in step 2408 withthe pixel values from different OCMs being grouped according to theimage area to which they correspond so that pixel values captured bydifferent OCMs but corresponding to the same image area can be combinedat a pixel level.

In some embodiments in addition to the pixel values from an opticalchain module, the processor has access to information about the lensspacing and/or configuration as well as the focus distance used by theindividual optical chain modules supplying the sets of pixel values forcombining. Thus, at least in some embodiments the processor has accessto spatial information which allows the processor to align pixels of animage captured by one OCM 161, 161′ or 161″ with that of another OCMwhich provides pixel values to be combined. Thus, the pixels can becombined based on the individual pixel size scene area to which theycorrespond. In other embodiments images captured by different OCMs canbe correlated based on content. The comparison of content allows pixelsof images captured by one OCM to be aligned for combining purposes withpixels captured by another OCM. As the result of the alignment of pixelvalues corresponding to the same scene areas but captured by differentOCMs, pixel values from different OCMs can be combined on a per pixelarea basis, e.g., with each OCM contributing, in some embodiments, atmost one pixel value to be used in generating a corresponding pixelvalue of the combined image.

Of course in other embodiments where area filters or other area basedfiltering is applied there may not be a one to one pixel correspondencebetween a pixel value provided by an OCM and a pixel in the combinedimage.

FIG. 25 illustrates a chart 2500 of the type which may be generated bystep 2408 with pixel values being arranged into sets corresponding todifferent pixel areas, with each row beginning with a different pixellocation identifier (P1, P2, P3, or P4) representing a set of pixelvalues corresponding to the same image area but captured by differentoptical chain modules. FIG. 25 will be discussed further below.

Operation proceeds from step 2408 to step 2410 in which the processorimplementing the method accesses exposure time information correspondingdifferent optical chains (OCM1 161, OCM 2 161′, OCM 3 161″) whichcontributed to pixel values in the set being processed. In step 2412 theprocessor proceeds to identify pixel values which correspond to sensorsaturation. Such values indicate that the maximum detection (e.g., lightcapture capability) was reached and that while the input was at least asstrong as indicated by the measured value it might be higher than themeasured value.

Operation proceeds from step 2412 to step 2414. In step 2414 pixelvalues which are deemed unreliable because of a saturation occurrenceare identified and excluded from further consideration. In the FIG. 24example, in step 2414 pixel values indicating that sensor saturationlevel was reached by an OCM with an exposure time longer than theshortest exposure time used by one of the OCMs providing pixel valueswhich are being combined are excluded from further consideration. Thisis because the pixel values from the OCM with the shorter exposure timemay provide more reliable information than the saturated values whichare being excluded.

FIG. 26 uses X's to indicate pixel values which will be excluded in oneembodiment as a result of performing step 2414 on a set of data such asthe one shown in FIG. 25. Operation proceeds from step 2414 to step 2418via connecting node A 2416.

In step 2418 a pixel value normalization operation is performed takinginto consideration the exposure times used by the different opticalchain modules. As should be appreciated the amount of light energydetected is normally a function of the exposure time with the amount ofenergy increasing proportionally to exposure time assuming that theimage does not change for the duration of the exposure. In step 2418pixel values to be combined are normalized based on exposure times with,e.g., pixel values corresponding to different OCMs being weighted basedon the exposure time used by the OCM supplying the pixel values. In someembodiments the weighting is based on the exposure time of the OCM fromwhich the pixel value was obtained and the shortest exposure time usedby an OCM to which some of the pixel values being combined correspond.

The results of pixel value normalization performed in step 2420 asapplied to the pixel values of FIG. 26 are shown in FIG. 27.

With the normalization of pixel values to be combined having beencompleted in step 2420, the pixel values can be combined in steps 2420on a per pixel location basis, e.g., through averaging or some otherstatistical method of combining values. FIG. 28 shows the results ofaveraging the pixel values of FIG. 27. As should be appreciated theresult of the combining operation is one pixel value for each of thepixel values corresponding to an image area obtained from the differentOCMS. For example, if three OCMs captured an image and provided pixelvalues corresponding to the same scene area, three or fewer pixel valuesmay be combined to generate a single pixel value of the final combinedimage.

Operation proceeds from step 2422 to step 2424 in which are check ismade to determine if there are sets of pixel values of another typeremaining to be processed. For example, if step 2422 produced a set of Rcombined pixel values, operation may proceed to step 2404 so that G or Bpixel values may be processed to generate corresponding pixel values forthe combined image. If in step 2424 it is determined that additionalsets of pixel values of a different type remain to be processed,operation proceeds to step 2404 via connecting node B 2436 so thatprocessing may proceed.

However, if in step 2428 it is determined that there are no additionalsets of pixel values corresponding to the image being generated to beprocessed, operation proceeds to step 2427 wherein the combined image isstored, e.g., in memory 108 or 1426, prior to the set of datarepresenting the combined image being output in step 2428. Outputting ofthe combined image may involve supplying the generated sets of R, G, Band/or luminance pixel values to a display device for rendering andpresentation on the display and/or may involve transmitting the pixelvalues representing the combined image over a communications channel,e.g., a network connection or broadcast channel, to supply one or moredevice with access to the network connection or channel with thecombined image generated from the pixel values captured by multipleoptical chain modules, e.g., modules 161, 161′, 161″.

The exemplary combining process shown and explained with reference toFIG. 24 can be applied and used in cases where OCMs all use the sameexposure time in which case all the pixel values from the OCMs which canbe correlated to the same image area can be combined to generate a pixelvalue of the combined image without concern or need for normalization ofpixel values assuming that the sensors used are the same in each of theOCMs. Averaging of pixel values from different sensors having the sameexposure time provides benefits with respect to averaging that helpreduce the effect that thermal noise or random differences in photonstrikes may have on the pixel value measured by different optical chainsfor the same area of an image. In essence using multiple OCMs reducesthe effect of noise and other random effects on the overall imagequality.

The exemplary method described with regard to FIG. 24 will now beexplained further with reference to the exemplary pixel values shown inFIGS. 25, 25, 27 and 28.

FIG. 25 illustrates an exemplary chart 2500 including a set of pixelvalues, e.g., luminance pixel values, generated by multiple opticalchains, e.g., a first optical chain (OCM 1), a second optical chain (OCM2) and a third optical chain (OCM 3) operating in parallel. The opticalchains OCM 1, OCM 2, OCM 3 for purposes of the FIG. 25 example are ofthe same type, e.g., unfiltered (luminance detection) optical chains.However, they could be red, green or blue filtered optical chain pixelvalues with all the pixel values corresponding to the same color. It isassumed for purposes of the example that the sensors of the opticalchains generate pixel values in the range of 0 to 255 with 0 indicatingdetection of no energy (e.g., a black image region) and 255 indicatingthe detection of the maximum amount of energy (e.g., a bright imageregion) the sensor can detect and that the sensors have the same orsimilar dynamic range, e.g., they have the ability to each measure thesame amount of energy before saturating. Thus, a pixel value of 255indicates saturation of a sensor, e.g., due to the sensor correspondingto a bright area of the image. The FIG. 25 chart is exemplary of thegrouping produced in step 2408 of the method shown in FIG. 24.

As should be appreciated, small sensors are often subject to saturationproblems do to their small size, e.g., they have a small bucket forstoring energy corresponding to received photons. With small sensorsthermal or other noise can also be an issue particularly in low lightconditions. As discussed elsewhere in the application thermal noise canbe improved by averaging pixel values captured by multiple opticalchains.

In the FIG. 25 example, the first OCM (OCM1) uses an exposure time of1/90th of a second, the second OCM (OCM 2) uses an exposure time of1/60th of a second and the third optical chain (OCM 3) uses an exposuretime of 1/30th of a second. For purposes of the example each of theexposures start at the same time. Thus, for at least a portion of thelight capture period, i.e., the first 1/90th of a second all three ofthe optical chains operate in parallel to capture light, during thesecond 1/90th of a second the second and third optical chains operate tocapture light and during the third 1/90th of a second only the thirdoptical chain operates to capture light. As should be appreciated thefirst optical chain, given its low exposure time, is particularly usefulin determining pixel values corresponding to very bright areas where theother sensors are likely to saturate due to their longer exposure times.The second optical chain is useful in covering a wide range of luminanceintensities but is not as good as the third optical chain which has alonger exposure time for capturing pixel values corresponding to imageareas which are low light, e.g., dark image areas. The third opticalchain is useful in providing information corresponding to low lightscene areas but is likely to saturate with respect to pixelscorresponding to high light image areas. It should be appreciated thatuse of the third optical chain with the long exposure time providesbenefits in terms of low light image regions while the first opticalchain provides benefits with respect to capturing pixel values in verybright image regions.

In the FIG. 25 chart 2500, an (S) is used under the pixel value toindicate that the value indicates saturation of the sensor. The firstcolumn lists pixels, e.g., pixels P1, P2, P3, P4, each pixelcorresponding to a different scene image area of a scene sizecorresponding to the sensor area of one pixel. While only four pixelsare included in the FIG. 25 example it is to be understood that an imagewill include, in many cases, millions of such pixels, each captured by adifferent portion of a sensor. In each row, the value captured by anoptical chain module corresponding to the pixel area of the scene isshown. For example in row 1, with regard to pixel area P1, the first OCM1 measured a pixel value of 100, OCM 2 measured a pixel value of 154 andOCM3 measured a pixel value of 255.

FIG. 26 is a chart 2600 illustrating the set of captured pixel valuesafter exclusion of pixel values corresponding to saturated pixel valuesmeasured by optical chain modules (OCM 2 and OCM 3) having longerexposure times than the optical chain module (OCM1) with the shortestexposure time which contributed to the set of pixel values shown in FIG.25. In FIG. 26 “X” is used to show saturated pixel values which areexcluded, in accordance with step 2414, from further use in generatingthe combined image.

FIG. 27 is a chart 2700 illustrating the normalized pixel valuesgenerated by processing the values shown in FIG. 26 with the excludedvalues being omitted. Note the pixel values shown in column 3corresponding to the second optical chain module (OCM2) are (⅔) thevalue of the original value reflecting that they were generated using1.5 times the shortest exposure time and that the values in the lastcolumn of the chart 2700 are ⅓ the original values reflecting that theywere generated using an exposure time three times the shortest exposuretime. The values in the first column corresponding to OCM 1 are leftunchanged since they correspond to the shortest exposure time andalready reflect the maximum energy per minimum image capture time (1/90th of a second) used by the optical modules in generating thecombined images.

The Normalized values remaining in the chart shown in FIG. 27 areaveraged in the particular exemplary embodiment to generate the pixelvalues of the combined image. FIG. 28 is a chart showing the resultingpixel values for pixels P1 though P4 of the combined image and thecomputation used to generate the pixel values. Note that the normalizedpixel values used to generate the combined pixel value for an image areacorresponding to the area of a pixel may be based on one, some or all ofthe outputs of multiple optical chains. This allows, in the case of lowlight conditions averaging which reduces the effects of noise and therandom nature of photon strikes in generated the pixel value as comparedto the case where the output of a single sensor may be used to generatethe combined pixel value. In the case of dark image regions, a pixelvalue from the optical chain module or optical chain modules with thelongest exposure time may be used. While the combined pixel valuecorresponding to a bright area may be generated from fewer sensors thanthat of low light areas, the higher energy level is less prone to theeffects of noise and random photon strikes often requiring multiplephotons to be sensed to produce the high pixel value. Accordingly, themethod shown in FIG. 24 provides benefits with respect to noisereduction where they are needed most, e.g., in the case of low lightareas of an image.

It should be appreciated that the method shown in FIG. 24 effectivelyincreases the useful dynamic range of the camera device including themultiple optical chains beyond that which could be achieved if all thesensors were exposed using the same exposure time where all the sensorswould saturate at or about the same light exposure level given that theywould be subject to the same exposure time.

While explained using an example with only one type of pixel values, itshould be appreciated that the method of FIG. 24 allows for images to begenerated using R, G and B pixel values which each of the R, G and Bpixel values being combined independently in some embodiments togenerate corresponding sets of R, G and B pixel values of a combinedcolor image.

The techniques of various embodiments may be implemented using software,hardware and/or a combination of software and hardware. Variousembodiments are directed to apparatus, e.g., a camera device, an imageprocessing device or a system. Various embodiments are also directed tomethods, e.g., a method of generating combined pixel values from sets ofinput pixel values corresponding to an image area where each set ofpixel values may be provided by a different optical chain module.Various embodiments are also directed to machine, e.g., computer,readable medium, e.g., ROM, RAM, CDs, hard discs, etc., which includemachine readable instructions for controlling a machine, e.g., cameradevice, processor or image processing system, to implement one or moresteps of one or more of the methods described in the presentapplication.

In various embodiments apparatus described herein are implemented usingone or more modules to perform the steps corresponding to one or moremethods. Thus, in some embodiments various features are implementedusing modules. Such modules may be implemented using software, hardwareor a combination of software and hardware. Optical chain modules asshould be appreciated include as least some hardware elements such as animage sensor and are therefore normally not implementable purely insoftware while other modules may be implemented fully in software. Insome embodiments in which the modules are implemented in hardware, themodules are implemented as circuits, e.g., of a processor and/or as acombination of hardware elements such as lenses, filters and an imagesensor. In many or all of the above described embodiments, methodsand/or method steps can, and in some embodiments are, implemented usingcomputer executable instructions, such as software, included in acomputer readable medium, e.g., a non-transitory computer readablemedium, such as a memory device, e.g., RAM, floppy disk, etc. which whenexecuted control a machine, e.g., general purpose computer or processor,with or without additional hardware, to implement all or portions of theabove described methods. Accordingly, among other things, variousembodiments are directed to a computer readable medium includingcomputer executable instructions for causing a machine, e.g., processoror computer system, to perform one or more of the steps of theabove-described method(s).

Some embodiments are directed to a processor configured to implement oneor more of the various functions, steps, acts and/or operations of oneor more methods described above. Accordingly, some embodiments aredirected to a processor, e.g., CPU, configured to implement some or allof the steps of the methods described herein. The processor may be foruse in, e.g., a camera device, an image processing device or other typeof system. In some embodiments the image processing device is a portabledevice including a camera, e.g., a cell phone including a camera with aprocessor that implements the method.

In some embodiments modules are implemented using software, in otherembodiments modules are implemented in hardware, in still otherembodiments the modules are implemented using a combination of hardwareand/or software.

Numerous additional variations on the methods and apparatus of thevarious embodiments described above will be apparent to those skilled inthe art in view of the above description. Such variations are to beconsidered within the scope of the invention.

What is claimed:
 1. A method of producing at least one image of a first scene area, the method comprising: operating a plurality of optical chain modules in parallel to capture images of the first scene area, the step of operating a plurality of optical chain modules including: operating a first optical chain module to capture a first image of said first scene area using a first color filter; and operating a second optical chain module to capture a second image of said first scene area using a second color filter, said first and second color filters corresponding to a first color and a second color respectively, said first and second colors being different colors; and processing said first and second images to generate a first combined image of said first scene area.
 2. The method of claim 1, wherein said first and second color filters are single color filters which correspond to said first and second colors, respectively.
 3. The method of claim 1, wherein the first optical chain module includes a first lens and a first image sensor; and wherein the second optical chain module includes a second lens and a second image sensor.
 4. The method of claim 3, wherein said first and second image sensors are of the same resolution.
 5. The method of claim 2, further comprising: operating a third optical chain module to capture a third image of said first scene area using a third color filter; and processing said third image to generate said first combined image of said first scene area from the third image in addition to said first and second images.
 6. The method of claim 5, further comprising: operating a fourth optical chain module to capture a fourth image of the first scene area using a multi-color filter; and displaying said fourth image on a display of a device including said first, second, third and fourth optical chain modules, said fourth image being displayed prior to capture of said first, second and third images.
 7. The method of claim 6, wherein said device is a camera.
 8. The method of claim 6, wherein said device is a mobile cellular phone.
 9. An apparatus, comprising: a plurality of optical chain modules that work in parallel to capture images of a first scene area, said plurality of optical chain modules including: a first optical chain module configured to capture a first image of said first scene area using a first color filter; and a second optical chain module configured to capture a second image of said first scene area using a second color filter, said first and second color filters corresponding to a first color and a second color respectively, said first and second colors being different colors; and an image processing module configured to process said first and second images to generate a first combined image of said first scene area.
 10. The apparatus of claim 9, wherein said first and second color filters are single color filters which correspond to said first and second colors, respectively.
 11. The apparatus of claim 9, wherein the first optical chain module includes a first lens and a first image sensor, and said second optical chain module includes a second lens and a second image sensor.
 12. The apparatus of claim 11, wherein said first and second image sensors are of the same resolution.
 13. The apparatus of claim 10, further comprising: a third optical chain module configured to capture a third image of said first scene area using a third color filter; and wherein said image processing module is further configured to process said third image in addition to processing said first and second images to generate said first combined image of said first scene area.
 14. The apparatus of claim 13, further comprising: a fourth optical chain module configured to capture a fourth image of the first scene area using a multi-color filter; and a display device configured to display said fourth image, said fourth image being displayed prior to capture of said first, second and third images.
 15. The apparatus of claim 14, wherein said apparatus is a camera.
 16. The apparatus of claim 14, wherein said apparatus is a mobile cellular phone.
 17. A computer readable medium including computer executable instructions which when executed by a processor control an apparatus to: operate a plurality of optical chain modules in parallel to capture images of a first scene area, said operating a plurality of optical chain modules including controlling a first optical chain module to capture a first image of said first scene area using a first color filter, and controlling a second optical chain module to capture a second image of said first scene area using a second color filter, said first and second color filters corresponding to a first color and a second color respectively, said first and second colors being different colors; and process said first and second images to generate a first combined image of said first scene area. 